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Between 1998 and 2004, Congress 
appropriated over $1.2 billion to 
the Office of National Drug Control 
Policy (ONDCP) for the National 
Youth Anti-Drug Media Campaign. 
The campaign aimed to prevent the 
initiation of or curtail the use of 
drugs among the nation’s youth. In 
2005, Westat, Inc., completed a 
multiyear national evaluation of the 
campaign. 

GAO has been mandated to review 
various aspects of the campaign, 
including Westat’s evaluation 
which is the subject of this report. 
Applying generally accepted social 
science research standards, GAO 
assessed (1) how Westat provided 
credible support for its findings 
and Westat’s findings about 
(2) attitudes, beliefs, and behaviors 
of youth and parents toward drug 
use and (3) youth self-reported 
drug use. 



What GAO Recommends 



Given that Westat’s evaluation 
stated the campaign did not reduce 
youth drug use nationally, 

Congress should consider limiting 
appropriations for the campaign, 
beginning in the 2007 fiscal year 
budget until ONDCP provides 
credible evidence of a media 
campaign approach that effectively 
prevents and curtails youth drug 
use. ONDCP’s written comments 
on our report generally disagreed 
with the findings. Specifically, 
ONDCP does not believe the 
Westat findings reflect the 
campaign’s effectiveness. We 
believe the Westat study is sound. 
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GAO’s review of Westat’s evaluation reports and associated documentation 
leads to the conclusion that the evaluation provides credible evidence that 
the campaign was not effective in reducing youth drug use, either during the 
entire period of the campaign or during the period from 2002 to 2004 when 
the campaign was redirected and focused on marijuana use. By collecting 
longitudinal data — i.e., multiple observations on the same persons over 
time — using generally accepted and appropriate sampling and analytic 
techniques, and establishing reliable methods for measuring campaign 
exposure, Westat was able to produce credible evidence to support its 
findings about the relationship between exposure to campaign 
advertisements and both drug use and intermediate outcomes. In particular, 
Westat was able to demonstrate that its sample was not biased despite 
sample coverage losses, maintained high follow-up response rates of 
sampled individuals to provide for robust longitudinal analysis, established 
measures of exposure that could detect changes in outcomes on the order of 
magnitude that ONDCP expected for the campaign and that could reliably 
measure outcomes, and used sophisticated statistical methods to isolate 
causal effects of the campaign. 

Westat’s findings on the effects of exposure on intermediate outcomes — 
theorized precursors of drug use — were mixed. Specifically, although 
sampled youth and parents’ recall of campaign advertisements increased 
over time, they had good impressions of the advertisements, and they could 
identify the specific campaign messages, exposure to the advertisements 
generally did not lead youth to disapprove of using drugs and may have 
promoted perceptions among exposed youth that others’ drug use was 
normal. Parents’ exposure to the campaign led to changes in beliefs about 
talking about drug use with their children and the extent to which they had 
these conversations with their children. However, exposure did not appear 
to lead to increased monitoring of youth. Moreover, the evaluation was 
unable to demonstrate that changes in parental attitudes led to changes in 
youth attitudes or behaviors toward drug use. 

Westat’s evaluation indicates that exposure to the campaign did not prevent 
initiation of marijuana use and had no effect on curtailing current users’ 
marijuana use, despite youth recall of and favorable assessments of 
advertisements. Although general trend data derived from the Monitoring the 
Future survey and the Westat study show declines in the percentage of youth 
reportedly using marijuana from 2002 to 2004, the trend data do not 
explicitly take into account exposure to the campaign, and therefore, by 
themselves, cannot be used as evidence of effectiveness. In Westat’s 
evaluation of relationships between exposure and marijuana initiation the 
only significant finding was of small unfavorable effects of the campaign 
exposure on marijuana initiation during some periods of data collection and 
in some subgroups. 
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Accountability * Integrity * Reliability 



United States Government Accountability Office 
Washington, DC 20548 



August 25, 2006 

The Honorable Christopher Bond 
Chairman 

The Honorable Patty Murray 
Ranking Member 

Subcommittee on Transportation, Treasury, the Judiciary, 

Housing and Urban Development, and Related Agencies 
Committee on Appropriations 
United States Senate 

Congressionally mandated under the Treasury and General Government 
Appropriations Act of 1998, 1 the National Youth Anti-Drug Media 
Campaign had the primary goals of preventing the initiation of drug use — 
particularly the use of entry-level drugs marijuana and inhalants — among 
the nation’s youth and stopping youth that have begun using drugs from 
continuing their use. Administered through the Office of National Drug 
Control Policy (ONDCP) and implemented in three phases, the campaign 
featured as its centerpiece a paid advertising effort in which campaign 
funds were used to purchase media time and space for advertisements that 
delivered anti-drug messages to the campaign’s target audience — youth 
aged 9 to 18 and their parents — through strategic placement of anti-drug 
advertisements on television and radio and in print media. In addition to 
the advertising, the campaign included community outreach, work with 
the entertainment industry to encourage the accurate depiction of the 
consequences of drug use, outreach to faith-based organizations, and work 
with youth organizations. The campaign’s first two phases, which ran from 
January 1998 through the summer of 1999, were pilot phases that focused 
primarily on informing the planning and development for phase III and 
included a 12-city pilot (phase I) and nationwide advertising (phase II). 
Phases I and II aimed to increase public awareness of anti-drug messages. 
Phase III of the campaign, which began in mid-1999, continued the 
nationwide advertising campaign begun during phase II and integrated the 
advertising with outreach efforts. From fiscal year 1998 through fiscal year 
2006, Congress appropriated over $1.4 billion to support the campaign. For 
fiscal year 2007, the President’s budget requested $120 million for the 
campaign, an increase over the fiscal year 2006 appropriation, to purchase 



'Pub. L. No. 105-61, 111 Stat. 1272. 
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additional media time and space to increase the reach and frequency of 
the campaign’s messages, which would restore appropriations to their 
fiscal year 2005 level. 

Congress first authorized funding for the campaign in fiscal year 1998 with 
the expectation that demonstrable changes in youth drug behaviors would 
be apparent within 3 years, and Congress required ONDCP to assess 
whether the campaign’s efforts have been effective in changing the drug 
use behaviors of America’s youth. ONDCP also indicated that it 
anticipated that it would take 2 to 3 years for the campaign to affect drug 
use behavior, although ONDCP also indicated that it was with the 
implementation of phase III of the campaign, beginning in mid-1999, that 
ONDCP expected to see improvements in anti-drug attitudes that would 
lead to decreases in youth drug use within 3 years. We previously reported 
that ONDCP’s evaluations of the first two phases of the campaign 
produced inconclusive results because of various evaluation 
implementation problems and limitations of the analyses used to support 
findings about effects during these pilot phases. 2 In particular, we noted 
that the impact evaluations of phases I and II did not adequately gauge the 
overall level of anti-drug awareness generated by the campaign — the 
principal outcome measure for these two phases — and we identified site 
selection problems, unknown parent response rates, low school response 
rates, and data analysis issues contributing to the inconclusive results. 

To implement the phase III evaluation, ONDCP entered into an 
interagency agreement with the National Institute on Drug Abuse (NIDA), 
which in turn awarded contracts to Westat, Inc., through June 2005 for 
$42.7 million to conduct the evaluation. Westat subcontracted with the 
Annenberg School for Communication at the University of Pennsylvania, 
and staff from Westat and Annenberg were coprincipal investigators for 



2 See: GAO, Anti-Drug Media Campaign: ONDCP Met Most Mandates, but Evaluations of 
Impact Are Inconclusive, GAO/GGD/HEHS-OO-153 (Washington, D.C.: July 31, 2000). We 
also reported on ONDCP’s use of consultants in the campaign in GAO, Anti-Drug Media 
Campaign: An Array of Services Was Provided, but Most Funds Were Committed to 
Buying Media Time and Space, GAO-05-175 (Washington, D.C.: Mar. 31, 2005). We also 
described the phase III evaluation in GAO, Program Evaluation: Strategies for Assessing 
How Information Dissemination Contributes to Agency Goals, GAO-02-923 (Washington, 
D.C.: Sept. 30, 2002). 
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the study. 3 Westat’s phase III evaluation covered the period from 
September 1999 through June 2004 and studied the impact of the 
effectiveness of the nationwide campaign in reaching its target audience; 
affecting youth beliefs, attitudes, intentions, and behaviors with regard to 
drug use; and affecting their parents’ beliefs and attitudes toward drug use 
and affecting parents’ behaviors associated with interacting with their 
children and monitoring their activities. 

Westat evaluated the campaign using a longitudinal panel survey — the 
National Survey of Parents and Youth (NSPY) — that aimed to measure the 
campaign’s effectiveness by assessing changes in various outcomes within 
individuals over time in relation to their exposure to campaign messages. 
Westat’s evaluation assessed the effect of exposure to the campaign on 
youth drug use and on several key intermediate outcomes — such as youth 
and parent attitudes toward and beliefs about drug use and parental 
involvement with their children — that were believed to influence youth 
drug use. In conducting the evaluation, Westat submitted several interim 
reports that were used in part to inform decisions about the direction of 
the campaign. Westat submitted a draft of the final evaluation report to 
NIDA in February 2005. 4 

Both Congress and ONDCP recognized the need for a separate evaluation 
of the campaign, because of limitations associated with existing national 
surveys of drug use. Congressional conferees acknowledged their 
intention to rely on the evaluation to gauge the impact of the campaign, 
and also indicated that if the campaign failed to show its effectiveness, 
they would be compelled to reevaluate the use of taxpayer money to 
support it. ONDCP acknowledged that existing national surveys of drug 
use would not be able to answer the critical question of whether changes 
in drug use behavior and attitudes were the result of the campaign. These 
surveys do not ask respondents about their exposure and reactions to the 
messages of the campaign that can then be linked to their drug-related 
attitudes and behavior. For example, in a 2001 report on youth drug use 
and the campaign, ONDCP officials noted that while national surveys of 



3 Hereafter, we refer to the contractor as “Westat,” and this implicitly includes Annenberg. 
In addition, a second subcontractor, the National Development and Research Institutes, 
Inc., provided expertise in developing drug use questions and assisted in preparing the first 
special topics report on trends in drug use. 

4 Westat and Annenberg jointly submitted to NIDA all evaluation reports except for the final 
report, which was submitted by Westat only. 
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youth drug use showed flattening or declining youth marijuana use in 1999 
and 2000 and these trends suggested that the campaign may be having the 
desired impact , it was necessary to await the results of the campaign’s 
independent evaluation before drawing any definitive conclusions 
regarding the campaign’s contribution to changes in youth drug use. 

In a committee report for the fiscal year 2004 appropriations cycle, the 
Senate Appropriations Committee directed us to review how consultants 
were used in support of the media campaign. 5 This is the second of two 
reports responding to this mandate. 6 The first report provided information 
concerning ONDCP’s use of consultants in the campaign. This second 
report addresses three questions related to Westat’s evaluation of phase III 
of the media campaign: (1) How did Westat ensure that it could report 
credible results in its evaluation of the campaign? (2) What did the 
evaluation find about the effect of exposure to the campaign on key 
intermediate outcomes that were intended to lower youth drug use? 

(3) What did the evaluation find about the effect of exposure to the 
campaign on youth drug use? 

In addressing our objectives, a team of GAO social scientists reviewed and 
assessed materials related to Westat’s phase III evaluation, applying 
generally accepted social science research standards, including such 
elements as when and how the sample data were collected, adjustments 
made to the sample to address nonresponse, how program effects were 
isolated (i.e., the use of statistical controls), and the appropriateness of 
outcome measures. The materials reviewed included interim and final 
evaluation reports, documentation and analyses provided by Westat to us 
in response to several sets of questions that we submitted about the details 
of its methodology, documentation pertaining to meetings of scientific 
panels that provided guidance on the evaluation, and documentation 
prepared by ONDCP about the design and implementation of the 
campaign. We conducted our work in accordance with generally accepted 
government auditing standards from October 2005 through June 2006. 



5 Senate Report No. 108-146, at 143 (2003). 

6 See GAO-05-175 for our review of ONDCP’s use of consultants in the campaign. 
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Results in Brief 



By collecting longitudinal data — i.e., multiple observations on the same 
persons over time — using generally accepted and appropriate sampling 
and analytic techniques, and establishing reliable methods for measuring 
campaign exposure, Westat was able to produce credible evidence to 
support its findings about the relationship between exposure to campaign 
advertisements and both drug use and intermediate outcomes. In 
implementing the study, Westat encountered problems that are common to 
large-scale longitudinal studies, and it addressed those using methods that 
are generally recognized as appropriate approaches for the study 
implementation challenges Westat faced. Challenges that Westat 
encountered were (1) lack of baseline data, which precluded Westat from 
comparing postprogram outcomes to preprogram conditions, and the 
redirection of the campaign; (2) sampling concerns, particularly ensuring 
the coverage of eligible households with youth in the targeted age range 
and ensuring that attrition over successive survey cycles did not result in 
insufficient sample size to detect campaign effects or in systematic bias 
within the sample; (3) establishing measures that would allow for both the 
sufficient detection of and the reliable measurement of exposure to the 
campaign on NSPY survey respondents; and (4) disentangling causal 
effects of exposure and drawing meaningful comparisons in the absence 
of ability to employ an experimental design where NSPY respondents 
would have been randomly assigned to various levels of exposure — 
the generally preferred approach for assessing program effects, where 
possible. Our examination of Westat’ s evaluation report and related 
documentation leads us to conclude that it addressed each of these 
challenges sufficiently to allow it to report credible findings about the 
effect of campaign exposure on drug use and intermediate variables 
believed to be precursors to drug use. Specifically, (1) several factors 
suggest that the lack of baseline data was not fatal to the evaluation’s 
findings, and Westat was able to generate statistically significant findings 
related to the redirected campaign; (2) Westat found no evidence of bias in 
the NSPY estimates despite sample coverage losses, and it also maintained 
high follow-up response rates of sampled individuals to provide for robust 
longitudinal analysis; (3) the NSPY sample could be used to detect 
changes in outcomes that were on the order of magnitude of changes that 
ONDCP expected for the campaign, and Westat demonstrated that its 
measures of exposure were valid and could reliably predict outcomes, 
whether results of the associations between exposure and outcomes were 
favorable or unfavorable to the campaign; and (4) using sophisticated 
statistical methods, Westat matched respondents on their underlying 
propensity to be exposed to campaign advertisements and, by comparing 
differences in outcomes among groups with different levels of exposure 
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resulting from its matching methods, isolated the effects of the campaign 
from other variables. (See appendix I for further details.) 

For intermediate outcome measures thought to influence the ultimate 
target of the campaign, youth drug use — for example, recall and 
identification of campaign messages, youth anti-drug attitudes, and 
parents’ beliefs and behaviors — Westat found favorable effects for some 
measures and subgroups, as well as unfavorable effects and no significant 
effects for others. In general, both youth and parents’ recall of specific 
campaign messages increased over the life of the campaign. In addition, 
NSPY trend data showed some increasing trends in anti-drug attitudes and 
beliefs as well as the proportion of youth who reported never intending to 
try marijuana. However, cross-sectional and longitudinal analysis provided 
no evidence that these trends resulted from campaign exposure. Westat’ s 
analysis also indicated that among current, non-drug-using youth, 
exposure to the campaign had unfavorable effects on their anti-drug 
norms and perceptions of other youths’ use of marijuana — that is, greater 
exposure to the campaign was associated with weaker anti-drug norms 
and increases in the perceptions that others use marijuana. Data for 
parents in the NSPY on five intermediate measures show some favorable 
effects of campaign exposure on parents’ behaviors and beliefs. However, 
for a major aim of the campaign, affecting parental behaviors regarding 
monitoring their children’s whereabouts, activities, and friends, Westat 
found no evidence of a significant effect. Moreover, where the data 
showed favorable relationships between campaign exposure and parental 
beliefs and behaviors, Westat did not find that these effects on parents 
ultimately lead to corresponding changes in their children’s beliefs and 
behaviors. 

Westat’ s evaluation found no significant favorable effects of campaign 
exposure on marijuana initiation among non-drug-using youth or cessation 
and declining use among prior marijuana users. Westat’ s NSPY data did 
show some declining trends in self-reported lifetime and past-month use of 
marijuana by youth over the period from 2002 to 2004 and declining trends 
in youth reports of offers to use marijuana. Declining drug use trends in 
the NSPY were consistent with trends in other national surveys of drug use 
over these years. However, Westat cautioned that because trends do not 
account for the relationship between campaign exposure and changes in 
self-reported drug use, trends alone should not be taken as definitive 
evidence that the campaign was responsible for the declines. ONDCP has 
also acknowledged the limitation of drug use trends for the purpose of 
demonstrating a causal link between campaign exposures and declines in 
drug use trends. Westat’s analysis of the relationship between exposure to 



Page 6 



GAO-06-818 Youth Anti-Drug Media Campaign 




campaign advertisements and youth self-reported drug use in the NSPY 
data for the entire period covered by its evaluation — assessments that 
used statistical methods to adjust for individual differences and control for 
other factors that could explain changes in self-reported drug use — 
showed no significant effects of exposure to the campaign on initiation of 
marijuana by prior nonusing youth. Westat’ s analysis found significant 
unfavorable effects — that is, a relationship between campaign exposure 
and higher rates of initiation — during one round of NSPY data and for the 
whole period of the campaign among certain subgroups of the sample 
(e.g., 12 Vi- to 13-year-olds and girls). Westat found no effects of campaign 
exposure on rates of quitting or use by prior users of marijuana. 

In light of the fact that the phase III evaluation of the media campaign 
yielded no evidence of a positive outcome in relation to teen drug use and 
congressional conferees’ indications of their intentions to rely on the 
Westat study, Congress should consider limiting appropriations for the 
National Youth Anti-Drug Media Campaign beginning in the fiscal 2007 
budget year until ONDCP is able to provide credible evidence of the 
effectiveness of exposure to the campaign on youth drug use outcomes or 
provide other credible options for a media campaign approach. In this 
regard we believe that an independent evaluation of the new campaign 
should be considered as a means to help inform both ONDCP and 
Congressional decision making. 

We provided a draft of this report to the Director of ONDCP for comment. 
In response, ONDCP provided written comments (reproduced in appendix 
II), which stated that ONDCP was puzzled that we did not make 
recommendations to it about how to improve the campaign. However, the 
main purpose of our report was to assess Westat’ s evaluation rather than 
to comment on how to improve the media campaign. In so doing, we 
focused on Westat’s methods. Our role was to inform Congress about the 
reliability of Westat’s evaluation so that Congress could decide the extent 
to which it will continue to fund the campaign. 

ONDCP expressed a number of concerns about our assessment of Westat’s 
evaluation and its implications concerning the effectiveness of the 
campaign. Most importantly, it stated that the Westat study is ill suited to 
assess impact and the study’s findings are of limited relevance. Our 
extensive review of the Westat study does not support ONDCP’s 
conclusion. Westat successfully addressed implementation challenges and 
used sophisticated analytic techniques to develop its findings. Another 
major issue ONDCP presents in its comments deals with the fact that the 
campaign has made major changes since the Westat data collection, 
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rendering the study’s findings irrelevant. Neither we nor ONDCP has 
factual data upon which to base an assessment of the effectiveness of the 
current campaign. However, other major efforts to substantially change 
the campaign during the time frame of the Westat data collection did not 
yield positive results. ONDCP raised a number of other issues that are 
generally related to the issues discussed above. These are addressed in the 
Agency Comments and Our Evaluation section of this report. 



Background 



The National Youth Anti- As part of the Treasury and General Government Appropriation Act of 

Drug Media Campaign 1998, 7 the Drug Free Media Campaign Act of 1998 required, among other 

things, the Office of National Drug Control Policy to conduct a national 
media campaign for the purpose of reducing and preventing drug abuse 
among young people in the United States. 8 The National Youth Anti-Drug 
Media Campaign may be the most visible federal effort devoted to 
preventing drug use among the nation’s youth. It aims to educate and 
enable America’s youth to reject illegal drugs; to prevent youth from 
initiating use of drugs, especially marijuana and inhalants; and to convince 
occasional users of these and other drugs to stop using drugs. 
Administered by ONDCP, and implemented in three phases, the campaign 
has as its centerpiece a paid advertising effort in which campaign funds 
were used to purchase media time and space for advertisements that 
delivered anti-drug messages to the campaign’s target audiences — youth 
aged 9 to 18 and their parents and adult caregivers — through strategic 
placement of anti-drug advertisements on television and radio and in 
print media. 

The campaign’s first two phases were pilot phases that had as their 
objectives developing advertising concepts, creating limited 
advertisements, testing public awareness of the advertisements in 
12 metropolitan areas, and eventually extending the pilot program 
nationwide. Phase III of the campaign, which began in mid-1999, continued 
the nationwide advertising campaign begun during phase II and integrated 



7 Pub. L. No. 105-61, 111 Stat. 1272. 

s Drug Free Media Campaign Act of 1998, 21 U.S.C. § 1801 et. seq. 
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the advertising with outreach efforts. In addition to the advertising, the 
fully integrated phase III campaign included community outreach, work 
with the entertainment and media industries to encourage the accurate 
depiction of the consequences of drug use, outreach to faith-based 
organizations, and work with youth organizations. 

During phase III, ONDCP had overall responsibility for developing and 
implementing the campaign, and to do so, it enlisted the support of 
nonprofit organizations, trade associations, private businesses, and federal 
agencies. Appropriated media campaign funds were to be used to cover 
the costs of actually making the advertisements as well as the costs for 
planning of purchase of media time and space. The campaign also 
included public outreach and specialized communications efforts. The 
purpose of public outreach and communications was to extend the reach 
and influence of the campaign through nonadvertising forms of marketing 
communications. Examples of these nonadvertising forms of 
communication included submitting articles related to key campaign 
messages such as effective parenting or the effects of marijuana on teen 
health to newspapers and magazines; building partnerships and alliances, 
for example, coordinating positive activities for teens with local schools 
and community groups; creating Web sites and exploring alternative media 
approaches; and entertainment industry outreach. 

According to the campaign’s communications strategy, youth aged 9 to 
18 were segmented into three school and age risk-level categories: late 
elementary school adolescents, aged 9 to 11; middle school children, aged 
11 to 13; high school youth, ages 14 to 18. The campaign originally targeted 
youth aged 9 to 18 with a focus on middle school age adolescents (roughly 
11-to 13-year-olds); its secondary focus was on high school-aged youth 
(approximately 14 to 18 years of age). In 2001, the campaign shifted its 
creative focus to 11- to 14-year-olds in order to more effectively reach 
youth at the time they are most at risk for trying drugs. In 2002, the 
campaign altered its target age group to focus primarily on 14- to 
16-year-olds. For all age groups, the communications strategy identified 
the primary focus of the campaign as at-risk nonusers and occasional 
users of drugs. For all groups, it was designed to give consideration to 
differences arising from gender, race, ethnicity, and regional and 
population density factors. 

From fiscal year 1998 through fiscal year 2004, Congress appropriated 
$1,225 billion to support the campaign (table 1). 
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Table 1: Appropriations for the National Youth Anti-Drug Media Campaign, Fiscal 
Years 1998 through 2006 


Fiscal year 


Final appropriations (in millions of dollars) 


1998 


$195 


1999 


$185 


2000 


$185 


2001 


$185 


2002 


$180 


2003 


$150 


2004 


$145 


2005 


$120 


2006 


$100 


Total, 1998 through 2006 


$1,445 


Total, 1998 through 2005 


$1,345 


Total, 1998 through 2004 


$1,225 



Sources: Appropriations Acts from various years. 



Note: Appropriated amounts are prerescission amounts. For example, rescissions were 0.38 percent 
in fiscal year 2000, 0.22 percent in fiscal year 2001 , 0.65 percent in fiscal year 2003, and 0.59 
percent in fiscal year 2004. 

For fiscal year 2007, the President’s budget requested $120 million for 
campaign activities. The 2007 request represents an increase of $21 million 
above the fiscal year 2006 budget authority. The additional resources were 
requested to help to purchase additional media time and space to increase 
the reach and frequency of the campaign’s messages. 



Planning and the 
Underlying Logic of the 
Campaign 



According to ONDCP, its planning for the campaign’s communications 
strategy included reviews of published studies on the etiology and 
prevention of adolescent drug use, drug prevention campaigns, other 
public health campaigns, and general consumer marketing campaigns 
targeting youth and their parents. ONDCP also supplemented its research 
evidence with an extensive expert consultation process that included 
input from over 200 experts in academia, civic and community 
organizations, government agencies, and the private sector. A campaign 
design expert panel that included experts in the fields of drug use and 
prevention, public health communication, advertising, market research, 
consumer marketing, and public policy met over a 4-day period during the 
fall of 1997 and played a key role in integrating diverse sources of 
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information and guiding the development of the communications strategy 
for the campaign. 

The planning process resulted in a statement of ONDCP’s communications 
strategy for the campaign, which described the premises of the campaign. 
Among these were the following: First, that the media can influence 
people in a variety of ways, such as informing and alerting them to 
important developments and shaping subsequent actions; satisfying leisure 
time needs, thereby influencing individuals’ views and beliefs about the 
world; and stimulating interest in commercial goods and services, thereby 
influencing where and how people shop. Second, that media messages 
have more potential to reinforce rather than to alter existing attitudes and 
beliefs. Third, to the extent that youth attitudes, beliefs, and intentions 
toward drug use vary with their age, the potential of a media campaign to 
influence drug use may be directly related to the age of the youth. Fourth, 
the campaign had to be sustained over time and to have a significant 
media presence, and its central messages have to be repeated often and in 
a variety of ways. Citing research showing that attitudinal and behavioral 
change took time to occur, ONDCP reported that it expected to observe 
“improvements in anti-drug attitudes that would lead to decreases in youth 
drug use within three years” of the implementation of phase III of the 
campaign. Fifth, as parents and adult caregivers play a vital role in youth 
drug use behaviors, and by also targeting parents, the campaign would aim 
to affect the nature of their interaction with their children, thereby 
strengthening their children’s capacity to resist using illicit drugs. 

The campaign focused on primary prevention — that is, preventing those 
who did not use drugs from starting to use drugs. According to ONDCP, a 
media campaign that focused on primary prevention targets the underlying 
causes of drug use and therefore has the greatest potential to reduce the 
scope of the problem over the long term. Further, a primary prevention 
campaign also has greater potential to affirm and reinforce anti-drug 
attitudes of nonusers than to persuade experienced users to change their 
behaviors, and a primary prevention campaign would also, over time, 
lessen the need for drug treatment services. With a focus on young, non- 
drug-using adolescents, an expectation underlying the campaign’s 
potential success was that as these young, non-drug-using adolescents 
aged, the campaign’s messages would intervene, retard the development of 
more pro-drug attitudes, and enable adolescents to continue to maintain 
their preexisting anti-drug attitudes. By maintaining these attitudes, or 
preventing the development of pro-drug sentiments, the campaign would 
affect drug use rates by lowering the rate at which youth initiated drug 
use, particularly the use of marijuana or inhalants. 



Page 11 



GAO-06-818 Youth Anti-Drug Media Campaign 




The campaign was designed to have a significant and sustained media 
presence. During planning, ONDCP acknowledged that the campaign 
would have to be sustained for a period of time sufficient to bring about a 
measurable change in the beliefs and behaviors of youth in the target 
audience. On the basis of the experiences of successful social marketing 
campaigns, ONDCP reported that it expected that changes in awareness or 
recall of the campaign would be detectable within a few months of the 
start of the campaign, that changes in perceptions and attitudes would be 
detectable within 1 to 2 years of the start of the campaign, and that 
changes in behavior would be detectable within 2 to 3 years. 



Campaign Activities during From mid-1999, the start of phase III, through June 2004, the end of the 
Phase III phase III evaluation, campaign activities included extensive media 

dissemination of campaign messages to a national audience of youth and 
parents; an interactive media component, which involved using content- 
based Web sites and Internet advertising; use of experienced individuals 
and organizations with expertise in marketing to teens, advertising and 
communications, behavior change, and drug prevention to inform the 
campaign strategy and implementation; use of multicultural initiatives that 
focused on sufficiently exposing campaign messages to African 
Americans, Asian Americans, Pacific Islanders, Hispanic Americans, 
American Indians, and Alaskan Natives; and the implementation of the 
integrated social marketing and public health communications campaign 
through the creation of partnerships with civic, professional, and 
community groups and outreach to media, entertainment, and sports 
industries. Through the partner organizations, the campaign attempted to 
strengthen local anti-drug efforts, and through outreach, it encouraged the 
news media to run articles that conveyed campaign messages. Youth and 
parent exposure to campaign messages could come from the direct, paid 
and donated advertising or from content delivered by news media and 
entertainment industries through the outreach efforts. Additional 
opportunities for exposure to anti-drug messages could be enhanced 
through personal involvement with organizations that became partners as 
a result of campaign outreach or by interaction with the campaign’s Web 
site. Further, youth exposure to anti-drug messages could also occur 
through interactions with friends, peers, parents, or other adults that 
occurred directly from either campaign ads or outreach efforts. 
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Campaign Themes and 
Messages 



Campaign messages for both youth and their parents and caregivers were 
to focus on common transitions — such as the transition from elementary 
to secondary school — and common situations — such as the amount of 
time spent in settings without adult supervision — that were believed to 
heighten adolescents’ vulnerability to drug use initiation. In addition, 
messages were to focus on altering mediating variables — such as beliefs 
and intentions — that were known to have a significant impact on 
adolescent drug use. Finally, campaign messages were designed to create 
a “brand identity” in the minds of target audience members and through 
brand identity position campaign messages as credible and important. 
Throughout phase III, themes such as parents as “The Anti-Drug” and the 
“My Anti-Drug” theme for youth were designed to promote identification 
and positive associations with the campaign’s messages. 

While they evolved throughout the campaign, the central strategic 
messages or themes for youth focused on resistance skills and self-efficacy 
to refuse drugs, normative education and positive messages, negative 
consequences of drug use, and early intervention. Resistance skills and 
self-efficacy advertisements were designed to enhance the personal and 
social skills of youth that promote lifestyle choices and to help build 
youth’s confidence that they could resist drugs. Normative education 
themes attempted to instill the beliefs that most young people do not use 
drugs or convey messages that “cool people don’t use drugs,” while 
positive message themes reinforced the idea of positive uses of time as 
alternatives to illicit drugs. Negative consequences themes aimed to 
enhance youth perceptions that drug use is likely to lead to a variety of 
negatively valued consequences, such as loss of parental approval, 
reduced performance in school, and negative social, aspirational, and 
health effects. Negative consequences themes were the primary focus of 
the Marijuana Initiative, which was introduced during 2002. An early 
intervention theme sought to motivate youth to intervene with friends who 
they perceived as having problems with drugs or alcohol and tried to 
convince youth of their ability to take action and to give them the tools 
and skills they needed to intervene. 

For parents, the campaign’s themes included messages that every child, 
including their own, was at risk of doing drugs; that they can learn 
parenting skills to help them help their children avoid drugs; that they 
need to be aware of the harmful effects of drugs including marijuana and 
inhalants; and, as part of the Early Intervention Initiative, that it was 
important that they intervene at the earliest possible opportunity in their 
child’s life if their child was using drugs or alcohol. 
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Design of the Evaluation, 
Interim Evaluation 
Reports, and Redirection 
of the Campaign 



ONDCP recognized the need for a separate evaluation of the campaign and 
for ongoing reporting of evaluation results. The need for a separate 
evaluation stemmed in part from the limitations of existing national 
surveys that monitor drug use, such as Monitoring the Future, which 
provides data on drug use by high school students, the National Household 
Survey on Drug Abuse, 9 and the Youth Risk Behavior Survey, which 
addresses health risk behaviors including drug use. These recurring 
surveys provide very little information with which to evaluate the impact 
of the campaign, because they were not designed to evaluate it. As ONDCP 
has written, these surveys contain no questions about target audience 
exposure and response to the campaign, and as a result, any changes in 
attitudes, beliefs, and behaviors toward drug use could not be associated 
directly with the campaign. By comparison, ONDCP acknowledged that it 
was using the Westat evaluation to assess the extent to which changes in 
anti-drug attitudes and beliefs or drug-using behavior can be attributed to 
the campaign, as opposed to other socioeconomic factors. In addition, 
ONDCP indicated that for the campaign, data from Westat’s evaluation 
would enable ONDCP to assess whether the campaign is working. 



The primary tool of the Westat evaluation was the National Survey of 
Parents and Youth. The NSPY is a longitudinal panel study of children and 
their parents’ exposure and response to the campaign. The NSPY was 
designed to collect initial and follow-up data from nationally 
representative samples of youth aged 9 to 18 and from the parents of these 
youth. The sample was designed to represent youth living in homes in the 
United States and their parents. Data collection began in November 1999 
and was conducted over four rounds — each of which was about 1 year 
apart from the next round — in nine waves of interviews. An interview 
wave refers to the fielding of a survey round to a specific subsample in the 
NSPY. An interview round refers to the completion of interviews with the 
entire sample. Data for each of the nine waves were collected using a 
laptop computer and a combination of computer-assisted interview 
technologies. To collect sensitive data, audio computer-assisted self- 
interview technology was used, allowing respondents to self-administer 
the questionnaire in total privacy. The final wave of data collection was 
completed in June 2004 (fig. 1). Eligible youth and parents were to be 
interviewed four times. 



9 The survey is now known as the National Survey on Drug Use and Health. 
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Figure 1: Data Collection Rounds and Waves of the NSPY 



Phase of data Survey Survey 

collection round wave 

Recruitment phase Round 1 Wave 1 



Data 

collection 

period 

11/99-5/00 



Cohorts (groups) of youth and parents 
interviewed in specific survey waves 



Group A, 

recruitment interview 





Wave 2 


7/00-12/00 




Group B, 

recruitment interview 




Wave 3 


1/01-6/01 


1 


r 


Group C, 

recruitment interview 


Follow-up phase 


Round 2 Wave 4 


7/01-12/01 


Group A, 
first follow-up 
interview 


1 


r Jr 





Wave 5 


1/02-6/02 


▼ 


Group B, 
first follow-up 
interview 


Group C, 
first follow-up 
interview 


Round 3 


Wave 6 


7/02-12/02 


Group A, 
second follow-up 
interview 


▼ 


▼ 




Wave 7 


1/03-6/03 


▼ 


Group B, 
second follow-up 
interview 


Group C, 
second follow-up 
interview 


Round 4 


Wave 8 


7/03-12/03 


Group A, 
third follow-up 
interview 


▼ 


▼ 




Wave 9 


1/04-6/04 




Group B, 
third follow-up 
interview 


Group C, 
third follow-up 
interview 



Source: Adopted from Westat, 2005, Vol. 2: Appendices. 

The evaluation aimed to assess whether exposure to the campaign 
affected the self-reported knowledge, attitudes, beliefs, and drug use of 
youth. Because the campaign reached out to all youth nationwide, the 
evaluators could not assess its effects using experimental methods, in 
which some subjects are randomly assigned to the intervention and others 
are randomly assigned to control groups that were not exposed to the 
intervention. Westat’s evaluation was designed to take into account the 
variation in self-reported exposure to the campaign messages and to 
assess how this variation in exposure was correlated with outcomes that 
the campaign intended to affect. To attribute changes in drug use attitudes 
and behaviors to the campaign, the evaluation was designed to assess 
exposure to the campaign and to compare differences in outcomes for 
groups of persons that were exposed to varying levels of the campaign’s 
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messages, and to use statistical controls to account for individual-level 
differences among survey respondents. 

Westat’s evaluation assessed youth self-reported drug use and 
intermediate outcomes — such as youth and parent attitudes and beliefs 
toward drug use and parental involvement with their children — that were 
believed to influence youth drug use. The evaluation of phase III 
addressed issues related to (1) whether the campaign was reaching its 
target populations, (2) whether the desired outcomes moved in favorable 
or unfavorable directions, (3) whether the campaign was influencing 
changes in the desired outcomes, and (4) what could be learned from the 
overall evaluation to support ongoing decision making for the campaign. 
These issues led to the five major objectives for the evaluation: 

• to measure changes in drug-related knowledge, attitudes, beliefs, and 
behavior in youth and their parents; 

• to assess the relationship between changes in drug-related knowledge, 
attitudes, beliefs, and behavior and self-reported measures of media 
exposure, including the salience of the measures; 

• to assess the association between parents’ drug-related knowledge, 
attitudes, beliefs, and behavior and those of their children; 

• to assess changes in the association between parents’ drug-related 
knowledge, attitudes, beliefs, and behavior and those of their children 
that may be related to the campaign; and 

• to compare groups of people with high exposure to other groups with 
low exposure. 

Westat submitted semiannual and special topic reports to NIDA, as the 
findings from these interim evaluation reports were to be used to support 
ongoing decision making for the campaign. Westat submitted the first 
semiannual report in November 2000. By December 2003, Westat had 
submitted six additional reports, four of which were labeled as semiannual 
reports, and the other two included a special report on historical trends in 
drug use and a 2003 report of findings. 10 Westat submitted its first draft of 
its final report to NIDA in February 2005. 

In addition to Westat’ s evaluation of the relationship between exposure 
and outcomes, Westat also prepared a report on the environmental 



10 A11 of these reports were submitted jointly by Westat and Annenberg. 
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context of the campaign. 11 In May 2002, Westat reported findings from this 
qualitative study of views of representatives from major national 
organizations and state prevention coordinators about the messages 
conveyed by the campaign and the role of the campaign as an organizing 
partner in helping to bolster local substance abuse prevention efforts. 
According to Westat, representatives felt that the campaign’s messages 
reinforced their own messages that encouraged youth to find healthy 
alternatives to drug use and to raise public awareness of the issue of illicit 
drugs among youth. Westat also reported that representatives were less 
enthusiastic about the role of the campaign as an organizational partner in 
helping with local substance abuse prevention efforts. 

In November 2002, Westat submitted its fifth semiannual report to NIDA. 

In it, Westat reported that it found little evidence that the campaign had 
direct, favorable effects on youth self-reported drug use between 2000 and 
2002. Specifically, Westat reported: 

“There is little evidence of direct favorable Campaign effects on youth. There is no 
statistically significant decline in marijuana use to date, and some evidence for an 
increase in use from 2000 to 2001. Nor are there improvements in beliefs and 
attitudes about marijuana use between 2000 and the first half of 2002. Contrarily, 
there are some unfavorable trends in youth anti-marijuana beliefs. Also there is no 
tendency for those reporting more exposure to Campaign messages to hold more 
desirable beliefs.” 12 



Westat further reported that there were unfavorable delayed effects of 
campaign exposure on subsequent intentions to use marijuana and on 
other beliefs. By delayed effects, Westat referred to the relationship 
between exposure to the campaign measured in one survey round having 
an effect on intentions or beliefs outcomes at a subsequent survey round. 
For parents, Westat reported that the evidence was consistent with 
favorable campaign effects, as it found that there were favorable changes 
for three of five parents’ belief and behavior outcome measures. However, 
Westat also reported that it found no evidence for favorable indirect 



n Westat, “Environmental Context of the National Youth Anti-Drug Media Campaign: 
Findings from In-Depth Discussions with Representatives of National Organizations and 
State Prevention Coordinators.” Report delivered to National Institute on Drug Abuse, 
National Institutes of Health, Rockville, Maryland, May 2002. 

12 Hornik, Robert, et al. Evaluation of the National Youth Anti-Drug Media Campaign: 
Fifth Semi-Annual Report of Findings, (Rockville, Maryland: Westat, November 2002), 
p. xi. 
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effects on youth behavior as the result of their parents’ exposure to the 
campaign. 

Congressional appropriators expressed concerns about the findings of 
Westat’s fifth semiannual report. In the conference report for fiscal year 
2003 omnibus appropriations, the conferees reported that they were 
“deeply disturbed by the lack of evidence that the National Youth Anti- 
Drug Media Campaign has had any appreciable impact on youth drug 
use.” 13 The conferees further acknowledged that while the evaluation 
conducted under NIDA’s auspices showed “slight and sporadic impact on 
the attitudes of parents, it has had no significant impact on youth 
behavior.” The conferees further acknowledged that while other surveys 
of youth drug use — such as Monitoring the Future, a survey of high school 
youth — showed recent declines in drug use, “the NIDA study was 
undertaken to measure the specific impact of the Media Campaign, not 
simply to gauge general trends,” and the conferees stated that they “intend 
to rely on the scientifically rigorous NIDA study to gauge the ultimate 
impact of the campaign” and to reevaluate the use of taxpayer money to 
support the campaign if the campaign continued to fail to demonstrate its 
effectiveness. 

In 2002, the strategy for the campaign was redirected. In the spring, the 
target age group of the campaign became 14- to 16-year-olds — youth who 
have higher rates of marijuana initiation than younger youth — from its 
original targeting of 11- to 13-year-olds. The shift to teens in the 14- to 
16-year-old range aimed to allow the campaign to more effectively reach 
youth during the time at which they are most at risk for trying drugs. 
ONDCP also required more rigorous copy test procedures of all television 
advertisements before they were aired, and ONDCP increased its oversight 
in guiding the development and production of advertisements. In October 
2002, ONDCP launched a new initiative called the Marijuana Initiative. 

This initiative contained more focused advertising to address youth 
marijuana use. In a hearing before the House Committee on Government 
Reform, ONDCP announced that it would reverse the ratio of campaign 
advertising expenditures directed to adults and youth, respectively. 
Previously, about 60 percent of expenditures were directed to adults and 
40 percent toward youth. Finally, during February 2004, it expanded the 
campaign’s communications goals to include the Early Intervention 
Initiative. This intervention was targeted toward both parents and teen 



13 Conference Report No. 108-10, at 1345 (2003). 
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friends, and ONDCP intended to use parental and peer pressure to stop 
drug and alcohol use among teens. 



Assessment of the 
Campaign by the Office of 
Management and Budget 
and ONDCP’s Current 
Approach 



To strengthen the linkages between resources and performance 
envisioned in the Government Performance and Results Act of 1993 
(GPRA), 14 the Office of Management and Budget (OMB) developed the 
Program Assessment Rating Tool (PART) to bring performance 
information into the executive budget formulation process. PART is 
designed to determine the strengths and weaknesses of federal programs 
by drawing upon available program performance and evaluation data so 
that the federal government can achieve better results. The PART 
therefore looks at factors that affect and reflect program performance, 
including program purpose and design; performance measurement, 
evaluations, and strategic planning; program management; and program 
results. Because the PART includes a consistent series of analytical 
questions, it allows programs to show improvements over time and allows 
comparisons between similar programs. 



OMB’s PART rating of the campaign addressed issues related to its 
purpose and design, strategic planning, program management, and 
program results and accountability. OMB indicated that the purpose was 
clear — giving ONDCP a 100 percent score on this factor — and it rated the 
campaign’s planning and management with scores of 67 percent and 
70 percent, respectively. In its assessment of ONDCP’s strategic planning, 
OMB noted that in response to its 2002 PART review, ONDCP revised the 
campaign’s logic model and significantly changed its long-term and annual 
performance measures. 

However, OMB’s assessment rating for the campaign was “results not 
demonstrated.” OMB indicated that its assessment of the campaign’s 
progress toward both the long-term goals and annual performance goals 
will be reviewed against the results of the NIDA-managed evaluation. OMB 
noted that while there is no federal program closely comparable to the 
campaign, evaluations of other health behavior change efforts found short- 
term effects after exposure to media. While acknowledging that a final 
assessment of the effects of the campaign awaited the final report from the 
NIDA-managed evaluation, OMB also indicated that “outcome data from 
the evaluation suggest little or no direct positive effect on youth behavior 



14 Pub. L. No. 103-62, 107 Stat. 285 (1993). 
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and attitudes attributable to the campaign to date. Perhaps some positive 
effect on parental attitudes/behavior but that has not yet translated into an 
effect on youth.” 

ONDCP has credited the campaign, along with a variety of collective 
prevention efforts, with contributing to “significant success in reducing 
teen drug use, as evidenced by the 19 percent decline from 2001 to 2005.” 

It has introduced a new youth brand approach to connect youth with 
aspiration themes. ONDCP also has indicated that while it awaits our 
formal assessment of the evaluation, that it will use existing national 
surveys to evaluate the campaign and suspend its request for proposals for 
a new evaluation contract. Specifically, ONDCP indicated that it would use 
the MTF survey to track improvements in perception of the risk of drug 
use — a predictor of lower drug use by youth — and it would use a special 
analysis of the PATS survey — the Partnership for a Drug-Free America’s 
Attitude Tracking Survey — data on anti-drug messages. According to the 
2005 data from MTF, there were no significant 1-year declines in marijuana 
use for youth in any grade levels, and while gradual declines in the upper 
grades continued, declines halted for youth in the 8th grade. Additionally, 
for 8th graders, perceived risk of marijuana use held steady, while for 
youth in 10th and 12th grade, there was an increase in perceived risk of 
marijuana use. 



Recent Research on the Two recently released studies have reported that exposure to the 
Effects of the Campaign in campaign was associated with changes in past-month marijuana use under 
Local Settings certain conditions for certain groups of students exposed to the campaign. 

In one of the studies, 15 45 South Dakota high schools and their middle- 
school feeder(s) were randomly assigned to three groups: (1) a basic 
prevention curriculum, (2) a group given this curriculum with booster 
lessons, and (3) a control group. All schools were exposed to the 
campaign during the fall of 1999 and spring of 2000. This permitted the 
researchers to test for a synergistic effect between exposure to the 
campaign’s anti-drug messages and participation in the school-based drug 
prevention curriculum. The sample of about 4,100 youth were asked how 
often they had seen anti-drug advertisements in recent months in five 
media outlets that were used by the campaign, and the researchers 



15 Longshore, Douglas, et al., “National Youth Anti-Drug Media Campaign and School-Based 
Drug Prevention: Evidence for a Synergistic Effect in ALERT Plus,” Addictive Behaviors, 
Vol. 31, (2006) pp. 496-608. 
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measured exposure to the campaign that indicated whether or not the 
adolescents reported seeing ads at least one to three times per week in any 
of the five media outlets. Consistent with Westat’s fifth interim report, the 
evaluation of the South Dakota drug prevention curriculum found no 
direct effects of exposure to the campaign on its sample of adolescents’ 
use of illegal drugs. However, the evaluation also found that marijuana use 
in the past month was significantly less likely among adolescents who 
received both the curriculum with booster lessons and weekly exposure to 
the campaign’s messages. In other words, neither the enhanced curriculum 
nor the campaign alone had a substantial effect on marijuana use in the 
absence of the other. In addition, this evaluation’s measure of exposure 
was based on weekly exposure, suggesting that the synergistic effect of 
the campaign observed in these South Dakota schools was based on the 
delivery of repeated messages. 

The second study used monthly random samples of 100 youth from the 
enrollment lists of 4th to 8th graders in the public schools in the spring of 
1999 in two moderate-sized communities — Fayette County (Lexington), 
Kentucky, and Knox County (Knoxville), Tennessee — over 48 months from 
April 1, 1999, through March 31, 2003. 16 The study period included 
advertisements aired under the campaign’s Marijuana Initiative. Students 
in the samples aged over time and were 13 to 17 years of age at the 
beginning of the Marijuana Initiative. Youth in the samples were measured 
on marijuana use during the past 30 days, as well as on their attitudes 
toward marijuana. Exposure to television and radio advertisements was 
measured by self-reported past-month exposure. The study found that 
among high-sensation-seeking youth — that is, youth who desire novel, 
complex, and intense sensations and experiences and who are willing to 
take social risks to obtain them — exposure to the first 6 months of the 
campaign’s Marijuana Initiative led to reductions in marijuana use. The 
study’s authors reasoned that the effects that they found for the Marijuana 
Initiative were consistent with an approach termed SENTAR (for 
sensation-seeking targeting), in which high-sensation-seeking youth are 
targeted with high sensation value messages to prevent risky behaviors. 



ltJ Palmgreen, Philip, et al., “Effects of ONDCP’s Marijuana Initiative Campaign on High 
Sensation-Seeking Youth.” Paper presented to the American Public Health Association, 
Philadelphia, Pennsylvania, December 2005. 
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Westat’s Evaluation 
Design, Use of 
Generally Accepted 
and Appropriate 
Sampling and Analytic 
Techniques, and 
Reliable Methods for 
Measuring Campaign 
Exposure Produced 
Credible Evidence to 
Support Its Findings 



Westat was able to produce credible evidence to support its findings about 
the relationship between exposure to campaign advertisements and both 
drug use and intermediate outcomes by employing a longitudinal panel 
design — i.e., collecting multiple observations on the same persons over 
time — using generally accepted and appropriate sampling and analytic 
techniques and establishing reliable and sufficiently powerful measures of 
campaign exposure. Westat encountered various challenges and threats to 
validity that are commonly associated with large-scale longitudinal 
studies, including lack of an opportunity to use experimental methods, 
lack of baseline data, and changes in campaign focus that were not timed 
with data collection; issues with ensuring adequate sample coverage and 
controlling for sample attrition over time; establishing measures that were 
sufficient to detect and reliably measure campaign effects; and 
disentangling causal effects without being able to employ an experimental 
design where subjects would have been randomly assigned to different 
levels of exposure. Our review of Westat’ s evaluation report and 
associated documentation leads us to conclude that the design and 
methodology used in its evaluation responded appropriately to these 
challenges, resulting in credible findings. 



Although Elements of the 
Campaign Limited Choices 
of Evaluation Designs and 
Affected Data Collection, 
Westat’s Design Was 
Rigorous and Provided a 
Means to Test for 
Campaign Effects 



The nationwide scope of the campaign precluded Westat from using 
experimental methods or obtaining baseline data, and the timing of the 
introduction of some new campaign initiatives limited some of the data 
available to evaluate them. However, Westat’ s longitudinal panel survey 
design provided a framework for developing strong evidence of within- 
respondent changes in outcomes over time as a result of exposure to the 
campaign. The consensus of a scientific panel convened by NIDA in 
August 2002 to review the evaluation was that Westat’ s use of a national 
probability sample to study change arising from the campaign was 
preferable as the “gold standard” to a study based on other alternatives, 
such as in-depth community-based studies of the mechanisms of change 
and campaign effects. Additionally, the theoretical underpinnings of 
behavioral change through advertising, along with statistically significant 
outcomes in some but not all groups, suggest that the absence of baseline 
data and introduction of new campaign initiatives did not invalidate the 
evaluation’s findings. Finally, despite the introduction of new campaign 
initiatives that were not timed with data collection cycles, Westat was able 
to assess change in the NSPY data and generate statistically significant 
findings using these data. 



Westat’ s longitudinal panel design was based on the premise that effects of 
exposure to the campaign on outcomes could be measured and detected 
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within individuals over time, after controlling for various other factors that 
could have influenced outcomes. The design called for measuring the 
same respondents up to four times to assess how the natural variation in 
exposure to the campaign correlated with campaign outcomes. Westat’s 
approach — an exposure (or dose)-response model — is based upon a 
premise that respondents’ recall of advertisements (exposure or dose) is 
related to outcomes (response). In two recent studies of the effects of the 
campaign on specific groups of youth in local areas, an exposure-response 
approach has been shown to be an effective method for detecting effects 
of the campaign in reducing youth drug use in local settings. One of the 
studies reported a synergistic effect of exposure to the campaign and a 
classroom-based drug prevention curriculum among 9th grade students in 
45 South Dakota high schools. The other study reported reductions in drug 
use during the period of the redirected campaign among high-sensation- 
seeking youth in schools in Knoxville, Tennessee, and Lexington, 

Kentucky. To assess the possibility of preexisting differences between 
groups of exposed youth and parents that might explain both the variation 
in exposure to the campaign and variation in outcomes, Westat included in 
the NSPY structured interview many questions on personal and family 
history, and it used the responses to control statistically for differences in 
attributes of respondents in order to attempt to isolate the relationship 
between exposure to the campaign and outcomes. 

The absence of baseline data — t hat is, precampaign data on outcomes — 
was beyond Westat’s control, as phase III of the campaign began before 
the first wave of data collection for the phase III evaluation began. The lag 
between the start of phase III of the campaign in mid-1999 and the 
completion of the evaluation’s first round of data collection — around mid- 
2001 — leaves open the possibility that there were effects of the campaign 
that occurred very early on in the campaign, prior to when Westat began 
data collection. Several factors suggest that the absence of pre-phase III 
baseline data was not fatal to the evaluation’s findings. First, if there were 
effects of the campaign that could not be detected because of the absence 
pre-phase III baseline data, those effects must have occurred very rapidly 
and then endured throughout the remainder of the campaign, from 1999 
through 2004. However, rapid changes in youth drug use were not 
observed in MTF data; rather, the overall trend in MTF past year drug use 
was flat between 1998 and 1999. Second, rapidly occurring effects were 
not expected by ONDCP in designing the campaign. As we reported in 
2000, and as ONDCP wrote in 2001, ONDCP believed that it would take 
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2 to 3 years for changes in drug use to be evident as a result of the 
campaign. 17 

Another campaign design factor that affected Westat’s evaluation was the 
implementation of new campaign initiatives, such as the Marijuana 
Initiative, which were implemented at times that officials at ONDCP 
considered to be important, and therefore they may not have coincided 
with planned data collection for the evaluation, nor should they 
necessarily have done so. For example, the Marijuana Initiative was 
implemented in October 2002, and the NSPY data available to evaluate 
outcomes during it were limited to three complete survey waves. For its 
longitudinal analysis of change during the Marijuana Initiative, Westat was 
limited to data from two survey waves. Despite these limitations, the 
evaluation produced data that enabled Westat to detect effects during the 
period of the Marijuana Initiative. 



Sample Coverage Issues 
Did Not Invalidate Westat’s 
Assessment of the 
Effectiveness of Exposure 
to the Campaign on 
Intermediate and Drug Use 
Outcomes 



During the enrollment phase of the NSPY, Westat experienced sample 
coverage problems, in that it enrolled — or rostered — a smaller percentage 
of households with youth in the targeted age range than would be 
expected based on comparable Current Population Survey (CPS) 
estimates — the data that Westat used to develop its expectations about the 
percentage of households having youth in the targeted age ranges. 
Coverage refers to the extent to which a sample is representative of the 
intended population on specified characteristics, and it is important 
because the omission of segments of the intended population from a 
sample — or undercoverage — can lead to biased results, in that omitted 
segments may differ in some important respect from those segments 
included. Westat estimated the extent of undercoverage in the NSPY to be 
about 30 percent as compared to the CPS estimates, and according to 
Westat and NIDA, the undercoverage arose during the stage of sampling in 
which Westat was developing rosters of households that were believed to 
contain youth in the target age range. At this stage, the survey rostering 
process required entry into the household, which may have led 
respondents in potentially eligible households to refuse to participate. 



Our review of Westat’ s documentation leads us to conclude that there was 
no evidence of biased results due to undercoverage and that the sample 
was sufficiently reliable both for the purposes of estimating changes over 



17 GAO/GGD/HEHS-00-153, (Washington, D.C.: July 31, 2000), p. 68. 
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time in individual outcomes and for assessing the effectiveness of 
exposure to the campaign on outcomes. Westat’s comparisons of the 
estimated population characteristics of the NSPY — such as race and 
ethnicity of head of household and race and ethnicity of youth in 
households — with the estimated population characteristics from the CPS 
show that they are generally similar. That is, the distributions of 
characteristics of eligible households with youth included in the NSPY 
were broadly consistent with a variety of corresponding distributions from 
the 1999 CPS. These comparisons suggest that the NSPY estimated 
population by race and ethnicity was similar to that of CPS. Westat also 
used multivariate modeling techniques to develop weighting adjustments, 
and it developed weights to adjust its sample for nonresponse that were 
reasonably effective in reducing nonresponse bias. 

An additional test for bias in a sample is to compare estimates derived 
from it with estimates on the same variable derived from another sample. 
If the NSPY results were biased, then one would expect that estimates 
derived from it would differ from estimates derived from unbiased 
samples. For example, if eligible households refused to participate in the 
NSPY because they contained teens with drug issues and as a result 
avoided participation at a higher rate than did households containing teens 
without drug issues, then these higher refusal rates by households 
containing teens with drug issues would lead to NSPY estimates of the 
percentage of youth reporting that they used drugs that were lower than 
those obtained from other, comparable national surveys. According to 
data provided by NIDA officials and our review of Westat’s final report, 
estimated self-reported drug use rates from the NSPY are comparable to 
estimates derived from other major surveys of drug use, such as the 
National Survey on Drug Use and Health. For example, in the NSPY, rates 
of past-month marijuana use among 12- to 18-year-olds were 7.2 percent in 
2000, 8 percent in 2001, 8.9 percent in 2002, and 7.9 percent in 2003. These 
rates were similar to those reported for 12- to 17-year-olds in the National 
Survey on Drug Use and Health (NSDUH) of 7.2 percent in 2000, 8 percent 
in 2001, 8.2 percent in 2002, and 7.9 percent in 2003. If youth with known 
drug use problems consistently opted out of both the NSPY and the 
NSDUH — a hypothesis that is not testable with the available data — then 
the estimates from both the NSPY and the NSDUH of the true prevalence 
of youth drug use would be biased underestimates. 
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Sample Attrition across 
NSPY Interview Rounds 
Was Sufficiently Low to 
Allow for Reliable 
Assessments of the Effect 
of Campaign Exposure on 
Outcomes 



As the NSPY was a longitudinal survey — in which eligible sample 
respondents were re-interviewed up to three times after their enrollment 
interviews — attrition was a concern with which Westat had to contend. If 
comparatively large numbers of sample respondents were not retained 
across successive rounds of the survey, the capacity of the NSPY to 
provide data to assess changes in outcomes in response to exposure over 
time would be greatly diminished. Further, if attrition was specific to 
certain groups, then the NSPY estimates would also be biased. 



For the purpose of estimating within-respondent changes in outcomes in 
response to changes in exposure across sample periods — the main use of 
the NSPY data — Westat achieved follow-up longitudinal response rates of 
between 82 percent and 94 percent for waves 4 through 9, the follow-up 
waves to the first three enrollment waves. The longitudinal response rate 
consists of two elements: (1) the percentage of prior survey respondents 
that are tracked and for whom eligibility is determined and (2) the 
percentage of those eligible that actually complete an interview. Across 
the three follow-up survey rounds, Westat tracked and determined the 
eligibility to participate in a follow-up survey of between 92 percent and 
96 percent of the youth and parents who completed a survey in the prior 
round. Of these, Westat obtained consent and completed extended 
interviews with between 94 percent and 96 percent of youth and parents 
for whom eligibility for a follow-up survey had been determined. 



In our view, Westat’ s follow-up response rates resulted in a sample that 
was sufficient to provide reliable findings about the effects of exposure on 
outcomes. In addition, Westat’ s nonresponse adjustment methodology 
compensated for effects of differential response rates related to the 
percentage of persons in certain age groups, of certain races and 
ethnicities, of those that owned homes versus rented, those that were 
U.S. citizens versus noncitizens, and those with incomes below the 
poverty level. 



The NSPY Data Could Be 
Used to Detect Reasonably 
Small Effects, and Westat’s 
Measurement of Exposure 
and Outcomes Were Valid 
and Could Detect Effects, 
if They Occurred 



The NSPY sample could be used to detect changes in outcomes that were 
on the order of magnitude of changes expected by ONDCP for the 
campaign, and its measures of exposure were valid and reliably predicted 
outcomes. In early meetings on the design of the evaluation of the media 
campaign, ONDCP officials reported that it had a specific Performance 
Measures of Effectiveness system and that the campaign was embodied 
within the first goal of the National Drug Strategy, which was to “educate 
and enable America’s youth to reject illegal drugs as well as the use of 
alcohol and tobacco.” Under this goal, ONDCP’s proposed targets for 
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reducing the prevalence of past-month use of illicit drugs and alcohol 
among youth from a 1996 base year — by 2002, reduce this prevalence by 
20 percent, and by 2007, reduce it by 60 percent. ONDCP officials 
identified other specific targets, again from the base year 1996 — by 2002, 
increase to 80 percent the proportion of youth who perceive that regular 
use of illicit drugs, alcohol, and tobacco is harmful; and by 2002, increase 
to 96 percent the proportion of youth who disapprove of illicit drug, 
alcohol, and tobacco use. To achieve a goal of 80 percent of 12th grade 
youth who perceive that regular use of marijuana is harmful would require 
increasing the 1996 baseline percentage of youth perceiving marijuana as 
harmful from 60 percent, as measured by MTF, or by about 3.3 percentage 
points per year from 1996 to 2002. Westat’s sample could be used to detect 
this amount of annual change in youth attitudes. 

In order to detect changes in outcomes due to exposure to the campaign, 
it also was necessary that Westat accurately measure and characterize 
exposure to the campaign. Westat provided evidence for the validity of its 
measures of self-reported exposure, and the evidence suggests that the 
measure of exposure was both valid and reliable. To measure exposure to 
the campaign for both youth and parents, NSPY interviewers asked 
respondents about their recall of anti-drug advertisements (general 
exposure) and their recognition of specific current or very recent 
television and radio advertisements (specific exposure). 18 To facilitate 
measures of recall, respondents viewed television and radio 
advertisements on laptop computers. Youth and parents were only shown 
or listened only to advertisements targeted to their respective groups. In 
addition, both youth and parents were asked some general questions about 
their recall of advertisements seen or heard in various media, including 
television, radio, newspapers, magazines, movie theaters, billboards, and 
the Internet. 

Westat’ s assessments of the validity of its measure of exposure to 
campaign advertisements confirm that the NSPY data were able to 
measure exposure. First, Westat examined respondents’ recall of 
campaign advertisements using “ringer” television advertisements — 
advertisements that never had appeared. According to Westat’s analysis of 
ringer advertisements, youth were more likely to recognize an 
advertisement as a campaign advertisement when presented with an 



ls Each respondent was presented ads that had been broadcast nationally in the 2 calendar 
months prior to the interview. 
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actual campaign advertisement than a bogus one. For example, a far lower 
percentage of respondents (11 percent) claimed to have seen a ringer, or 
bogus, advertisement than the percentage who claimed to have seen the 
broadcast advertisements (45 percent), particularly the advertisements 
that were delivered with high frequency. The result held for youth and for 
parents. 

Second, comparing data on advertisement time purchases with self- 
reported exposure to these advertisements in the NSPY, Westat found a 
high correlation between advertising and exposure. Specifically, on the 
basis of analysis of individual advertisements’ gross rating points (GRP) — 
a measure of the underlying reach and frequency of each advertisement — 
and self-reported exposures by respondents, Westat found a high 
correlation between GRPs purchased by the campaign and self-reported 
exposure to advertisements among youth. The correlation for parents was 
somewhat smaller, but was also significant. Third, Westat also compared 
self-reported exposure with recall of the correct brand phrase and found a 
strong association between self-reported exposure and correct recognition 
of the brand phrase. This is further evidence for the validity of its 
measures of self-reported exposure. 

Westat measured a variety of outcomes for youth and parents and took 
steps to ensure that the measures were consistent with existing research. 
The youth questionnaires included numerous questions that were designed 
to measure exposure to the campaign advertisements and other anti-drug 
messages. The youth question areas included exposure propensity to 
media; current and past use of tobacco, alcohol, marijuana, inhalants, and 
Ecstasy; past discussions with and communication of anti-drug messages 
from parents and friends; expectations of others about respondent’s drug 
use; knowledge and beliefs about the positive and negative consequences 
of drug use; exposure to campaign messages; family and peer factors; 
personal factors; and demographic information. Westat used separate 
questionnaires for youth of different ages; one questionnaire was used for 
children (aged 9 to 11) and another one was used for teens (aged 12 to 18). 
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Westat’s Analytic Methods 
Aimed to Isolate Causal 
Effects of the Campaign 
and Did So Using 
Sophisticated Techniques 
That Enhanced the 
Strength of Its Findings 



In it analysis, Westat used three types of evidence to draw inferences 
about the effects of the campaign: (1) trend data — data that describe 
increases or decreases in drug use and other outcomes over time; 

(2) cross-section analysis — measures of association between exposure to 
campaign messages and individual drug use beliefs, intentions, and 
behaviors, at the time data were collected; and (3) longitudinal analysis — 
measures of association, for youth and parents who were observed at two 
points in time, between exposure to campaign messages at the earlier time 
on outcomes at the later time. 19 Westat indicated that trends over time, by 
themselves, could not be used to provide definitive support for campaign 
effects. Rather, the trends needed to be supported by measures of 
association. Westat also indicated that measures of association, whether 
cross-sectional or longitudinal, needed to control for variables that could 
influence outcomes independently of the campaign or otherwise confound 
the association between exposure and outcomes. Cross-section 
association between exposure and outcomes measured at the same time 
would provide stronger evidence of campaign effects than would trend 
data alone, provided that controls for other variables were introduced into 
the associational analyses. However, even if cross-section associations 
between exposure and outcomes hold after controlling for the effects of 
other variables, as Westat pointed out, there may remain an alternative 
explanation for cross-section associations: For example, an outcome — like 
perceptions of others’ use of drugs — may be the cause of exposure rather 
than an effect of it. Westat’s longitudinal analysis attempts to address the 
ambiguities that exist with cross-sectional associations. With longitudinal 
data, if, after controlling for other confounding variables, exposure 
measured at an earlier time is associated with an outcome at a later time, 
the inference made is that the causal direction is from exposure to 
outcome, since an effect cannot precede a cause in time. 



As the campaign was implemented nationally and it was therefore not 
possible to assign youth and their parents randomly to treatment and 
control groups, a major threat to the validity of the conclusions from the 
evaluation is that the observed correlations between exposure to the 
campaign and self-reported attitudes and behaviors could reflect 
preexisting differences among individuals in their underlying susceptibility 
to campaign messages. The evaluation’s associations between exposure to 
the campaign and self-reported initiation of marijuana use took into 
account statistically the individual differences in attributes among youth 



19 Westat also called its longitudinal analysis a “delayed effects” analysis. 
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who were exposed to various levels of campaign messages, and they 
adjusted for the influence of other variables that could determine 
marijuana initiation — called confounder variables. As such, Westat’s 
evaluation of the associations between campaign exposure and marijuana 
initiation have accounted for individual differences among youth and can 
be viewed as comparisons of outcomes for statistically similar individuals. 
Further, the statistical test Westat used in assessing the relationship 
between exposure and initiation did not rely upon assumptions of linearity 
between levels of exposure and initiation. Instead, it tests for an ordered 
relationship between exposure and an outcome such as marijuana use 
initiation. 

Westat used statistical methods to address the possibility that preexisting 
differences between individuals could have caused both reported levels of 
exposure and respondent outcomes, and its use of these methods 
contributed to the validity of its findings about the effects of the campaign 
on outcomes. If, independently of the campaign, individuals differed in 
their underlying tendencies to accept and recall campaign messages, and if 
the individuals who were more likely to recall advertisements also were 
those who were more likely to respond to advertisements, then, absent 
efforts to address this confounding factor, the findings about the 
evaluation would be questionable. This type of bias is often called a 
selection effect. If selection effects occurred in the campaign, then both 
exposure and reported changes in attitudes and behaviors could reflect 
underlying beliefs that were not affected by the campaign, despite the 
presence of statistical correlations between self-reported exposure and 
changes in attitudes and behaviors. 

To control for selection effects and the many factors that could have 
influenced both exposure and outcomes independently of, or in 
conjunction with, the campaign, Westat used propensity scoring methods. 
These methods limit the influence of preexisting differences among 
exposed groups by controlling for a wide range of possible confounding 
variables. Propensity score methods are used to create comparison groups 
that are similar on measured and potentially confounding variables but 
that differ on their levels of treatment. In the evaluation of the campaign, 
the comparison groups were similar on confounding variables but differed 
on their level of exposure to campaign messages. Propensity score 
methods replace a set of confounding variables with a single function of 
these variables, which is called the propensity score. In Westat’s analysis, 
an individual’s propensity score is considered to represent an individual’s 
probability of being assigned to a particular level of exposure to the 
campaign, conditional upon the individual’s values of the confounding 
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variables. By including relevant, potentially confounding variables and 
matching individuals on their propensity scores, Westat was able to 
minimize bias due to selection effects. The comparison groups that Westat 
created by using propensity score methods can be considered as statistical 
analogues to randomly assigning individuals to different levels of 
exposure. After creating these groups, Westat then analyzed outcomes 
among the groups having different propensities to be exposed to campaign 
messages. 

Our assessment of Westat’s methods leads us to conclude that Westat took 
reasonable steps to develop valid propensity models, and as a result of its 
models, its analysis identified the effects of the campaign, net of other 
factors included in its propensity score models. First, rather than simply 
compare individuals who were exposed to campaign messages with those 
who were not exposed, Westat estimated and compared groups of 
individuals with different levels of exposure, where the number of 
exposure groups was measured alternatively as a three- or four-level 
variable — e.g., low, medium, or high exposure . 20 Second, for the results of 
propensity methods to be valid, it is important that the propensity scoring 
models include all relevant variables that could otherwise explain 
differences in both exposure and outcomes, as evaluators can adjust only 
for confounding variables that are observed and measured. If an important 
variable is omitted from the propensity model, the results of analyses may 
be affected. Westat’s models included many relevant and potentially 
confounding variables. For example, in the youth models, the propensity 
score models included measures of demographic attributes, educational 
attainment and educational aspiration, family and parent background, 
parental consumption of television and other media, income and 
employment, reading habits, Internet usage, location of residence in an 
urban area, among other variables. Third, for propensity models to remove 
the effects of confounding variables from the association between 
exposure and response, it is necessary that the population means of the 
confounder variables not vary across exposure levels. If a confounder is 
successfully balanced, then it will have the same theoretical effect across 
all exposure levels. After estimating models, Westat also assessed and 
demonstrated the balance of variables in its propensity models. 



20 Propensity score methods have been demonstrated to be robust against bias associated 
with the specification of incorrect functional forms — e.g., linear rather than quadratic — 
of variables. 
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The Phase III 
Evaluation Provided 
Mixed Evidence of the 
Campaign’s 
Effectiveness on 
Intermediate 
Outcomes, but It 
Found No Effect of 
the Campaign on 
Parental Monitoring 
of Youth 



Westat reported mixed evidence about the effectiveness of the campaign 
on intermediate outcome measures — such as recall and identification of 
campaign messages, youth anti-drug attitudes, and parents’ beliefs and 
behaviors — that were thought to be causal factors influencing youth drug 
use, the ultimate target of the campaign. Most parents and youth recalled 
exposure to campaign anti-drug messages, and for both groups, recall 
increased during the September 1999 to June 2004 period covered by the 
phase III evaluation. For current, non-drug-using youth — whose resistance 
to initiating marijuana use the campaign intended to affect — although 
NSPY data showed some favorable trends in anti-drug attitudes and beliefs 
and in the proportion of youth who said that they would definitely not try 
marijuana, there was no evidence that exposure to the campaign 
influenced these trends. Conversely, among current, non-drug-using youth, 
evidence suggested that exposure to the campaign had unfavorable effects 
on their anti-drug norms and perceptions of other youths’ use of 
marijuana — that is, greater exposure to the campaign was associated with 
weaker anti-drug norms and increases in the perceptions that others use 
marijuana. On three of five parent belief and behavior outcome 
measures — including talking with children about drugs, doing fun 
activities with children, and beliefs about talking with children — the 
evidence pointed to a favorable campaign effect on parents. However, 
while there was mixed evidence on the effect of the campaign on parents’ 
beliefs and attitudes about monitoring children’s behaviors, there was no 
evidence to support a claim that the campaign actually affected parents’ 
monitoring behaviors — an area of the campaign’s focus for parents — and 
there was little evidence for favorable indirect effects on youth behavior 
or beliefs as the result of parental exposure to the campaign. 



Youth and Parents’ Recall 
of Campaign 

Advertisements Increased 
over Time, Their 
Impressions of the 
Advertisements Were 
Favorable, and They Could 
Identify the Campaign 
Brand 



According to Westat, the campaign purchased enough advertising time 
over the 58-month period from September 1999 to June 2004 to achieve an 
average exposure of 2.5 youth-targeted ads per week for youth and an 
average of 2.2 parent-targeted advertisements per week for parents. 
Westat’s estimates include campaign advertisements intended for either all 
youth or all parents, but they do not include exposure of youth to parent 
advertisements or parents to youth advertisements, nor do they account 
for separate advertising targeted to specific race- or ethnicity-defined 
audiences. 

Using exposure indexes, Westat measured trends in general and specific 
exposure to campaign advertisements. The general exposure index was 
based on questions that asked about exposure to anti-drug messages in 
recent months through a variety of channels, including movies, television, 
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radio, and billboards, and was not limited to campaign advertisements. 21 
The specific exposure index was based on recall of specific 
advertisements broadcast during the 60 days prior to the respondent’s 
interview, and was limited to advertisements that targeted the respondent. 
For example, for youth, only youth advertisements were sampled to 
measure specific exposure. Youth aged 12 '/a to 18 and their parents 
reported increasing levels of recall of specific but not general exposure to 
campaign advertisements over time. For both parents and youth, there was 
a sharp increase over time in the recall of specific exposure of television 
ads across the campaign. Westat speculated that the increase in specific 
recall may have arisen from better-placed, more memorable, or longer- 
aired advertisements rather than only to an overall increase in television 
advertisements. However, recall of all general anti-drug advertising was 
fairly stable over time, as there was no overall detectable change in 
reported general exposure over the course of the campaign. 

Beginning in 2001, when the evaluation started to measure brand phrase 
recall, and continuing through 2004, the evidence indicates that youth, in 
particular, exhibited increases in brand phrase recall. Advertising 
campaigns may use a brand phrase to provide a recognizable element, and 
to the extent that the brand is recognized and positively regarded, its 
familiarity may lead to a positive response to a new advertisement or 
increase the perception that each advertisement is part of a larger 
campaign. The campaign included both a parent and a youth brand. Brand 
messages may have involved a series of phrases or the portrayal of an 
activity or lifestyle as positive (e.g., participating in team sports) to set up 
the brand phrase of “The Anti-Drug.” Westat reported that the evidence 
from the NSPY shows that the greater the exposure to media campaign 
advertising, the more likely respondents were to recall the brand phrase. 

In addition, the more that respondents recalled specific ads, the more 
likely they were to recognize the brand phrase, although over time even 
those with less exposure had learned the brand phrase. 

Overall, youth reported favorable impressions of the subset of campaign 
television advertisements that they were asked to evaluate, and their 
favorable impressions increased over time. Responses to the 
advertisements — whether they were attention getting, convincing, or said 



21 According to Westat, the reference period for the general exposure index, is “in recent 
months,” and this wording was chosen to maintain equivalence to the wording used in the 
Monitoring the Future surveys in its questions about anti-drug advertising. 
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something important to the respondent — were positive among both youth 
and their parents. Parents’ evaluations of the advertisements were 
generally more positive than those of youth, and parents’ positive views 
also increased over time. 

In addition to distributing messages directly in media advertisements, the 
campaign aimed to reach its target audiences indirectly through other 
institutions and routes, such as community groups, in-school and out-of- 
school anti-drug education, and discussions among youth and parents, and 
youth and friends, concerning drug use and the drug advertisements. The 
NSPY data indicated that the campaign’s messages were not accompanied 
by similar increases in exposure to messages from other sources. Both 
youth and parents reported receiving anti-drug messages from other 
sources, but they did not consistently report increases in exposure to 
messages from these sources. For example, from the 2000 to 2004 samples, 
the percentages of youth reporting receiving in-school drug education 
messages and attending out-of-school drug education both declined. 



Westat Found That the 
Campaign Generally Had 
No Effect on the Attitudes 
of Youth Not Using 
Marijuana toward Its Use 
but That Exposure to the 
Campaign Was Associated 
with Unfavorable Effects 
on Youth Perceptions of 
Others’ Use of Marijuana 



Westat generally found no significant effects of campaign exposure on the 
cognitive outcomes of adolescent nonusers of marijuana — 
i.e., development of anti-drug attitudes and beliefs. For current nonusers, 
the evaluation reported on four cognitive measures and a fifth measure of 
their perceptions of others’ use of marijuana. Three of the four measures — 
attitudes and beliefs about the consequences of marijuana use; perceived 
social norms or pressures from parents, friends, and peers about 
infrequent or regular marijuana use; and perceived self-efficacy to avoid 
using marijuana, or their confidence to turn down use of marijuana under 
various circumstances — were premised to affect the fourth — youth 
intentions to use marijuana at all during the next year. The fifth outcome, 
perceptions of other youths’ use of marijuana, 22 was included to examine 
whether exposure to the campaign was leading to increased perception 
among youth that others use marijuana, and whether this perception, in 
turn, affected their own behaviors. 

Westat reported that the evidence from the analysis of trend data from 
2000 to 2004 for two of the youth cognitive measures — attitudes and 
beliefs about the consequences of marijuana use and intentions to use 
marijuana — showed significant increases in youth believing that marijuana 



“This was measured as the “Percent perceiving few other kids regularly use marijuana.” 
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use had negative consequences and significant increases in the percentage 
of youth that reported that they had no intention to use marijuana. 
However, evidence from both cross-section and longitudinal associations 
between exposure and these two cognitive outcomes did not substantiate 
that the favorable trends arose from exposure to the campaign. 
Specifically, the cross-sectional associations between both general and 
specific exposure to the campaign and intentions not to use marijuana 
show no significant favorable effects of exposure on this outcome. None 
of the cross-section associations between either general or specific 
exposure and intention to use marijuana are significant, and none of the 
longitudinal associations between specific exposure and intentions are 
significant. Two of the longitudinal associations between general exposure 
and intentions are significant, but the direction of the effect is unfavorable, 
in that greater exposure led to declines in intentions not to use marijuana. 
The evidence from the associational analyses between exposure and 
attitudes and beliefs about the consequences of marijuana use generally 
did not show an effect of the campaign. While there was one significant 
cross-section association between general exposure and attitudes and 
beliefs about consequences during the final two waves of survey data, 
there were no significant cross-section associations between specific 
exposure and attitudes and beliefs about consequences, nor were there 
any significant longitudinal associations with either general or specific 
exposure. 

The associational analysis also produced some evidence of unfavorable 
effects of exposure on social norms — i.e., social pressures from parents, 
peers, and other important persons about marijuana use. Westat’ s cross- 
section associations showed no significant effects of exposure on social 
norms, but its longitudinal associations showed that across all survey 
rounds, there was a significant relationship between specific exposure and 
weaker social norms. Westat’ s analysis of associations between exposure 
and perceptions of others’ use of marijuana also produced significant 
results. Cross-section associations between specific exposure and 
perceptions of others’ use were significant, as were longitudinal 
associations of this relationship. In other words, among youth who 
reportedly did not use marijuana at the time of their interview, there was a 
significant effect of specific exposure on the perception that others used 
marijuana, and the direction of the effect was unfavorable — that is, those 
reporting higher exposure to anti-drug ads were more likely to believe that 
their peers used marijuana regularly. A significant and unfavorable 
relationship between specific exposure and perceptions of others’ use of 
marijuana was obtained for the data covering the entire period of the 
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evaluation as well as for the period of the redirected campaign, from 
2002 to 2004. 



The Evaluation Reported 
Favorable Effects of the 
Campaign on Three Parent 
Outcomes but Not on 
Parental Monitoring 



A theme of the campaign was to encourage parents to engage with their 
children to protect them against the risk of drug use, and parent skills 
were a focus of parent advertising almost since the start of the campaign. 
The campaign encouraged parents to monitor their children’s behavior by 
knowing where they were and with whom, and to make sure that they had 
adult supervision. It also encouraged parents to talk with their children 
about drugs and to a lesser degree to engage in fun activities with their 
children. The evaluation observed five outcomes for parents, and for four 
of the five found significant and favorable effects of exposure to the 
campaign. For three outcomes — parent-child conversations about drugs 
(talking behavior), parents’ beliefs and attitudes about talking with their 
children about drugs (talking beliefs), and parents’ engagement with their 
children in in-home and out-of-home activities (fun activities) — both cross- 
section and longitudinal associations between exposure and outcomes 
were generally significant and favorable to the campaign. For parents’ 
beliefs and attitudes toward monitoring their children’s behaviors, Westat 
reported favorable trend and cross-sectional associations but no 
significant overall longitudinal effects of either general or specific 
exposure on this outcome. For the fifth outcome, parent monitoring 
behaviors — that is, parents’ knowing or having a pretty good idea about 
what their child was doing or planned to do — the evidence did not support 
a finding of an effect of the campaign. There were no significant favorable 
trends in parents’ reports of monitoring behaviors, and there were no 
significant cross-section or longitudinal associations of either general or 
specific exposure on monitoring behaviors. 



No Evidence of Favorable 
Effects of the Campaign on 
Youth Outcomes through 
Campaign Effects on 
Parental Outcomes 



Despite evidence of some favorable parental outcomes for the campaign, 
Westat found no significant evidence for the overall evaluation that these 
favorable parent outcomes affected youth attitudes and behaviors toward 
drug use. Specifically, for the entire period covered by the evaluation, 
Westat found no evidence of overall, indirect campaign effects on parents 
leading to changes in marijuana use, intentions to use marijuana, social 
norms, self-efficacy, or cognitions among youth who were not marijuana 
users. Westat found that there were some significant indirect effects of 
parental specific exposure on some youth outcomes for some subgroups. 
For example, parental specific exposure was favorably associated with 
intentions to use marijuana for 14- to 18-year-olds and for boys, and it was 
also associated favorably with attitudes and beliefs about the 
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consequences of marijuana use for Hispanics. Westat also found 
significant but unfavorable indirect effects of parents’ general exposure on 
subgroups of youth in other youth outcomes. For example, parental 
general exposure was unfavorably associated with youth social norms for 
14- to 16-year-olds and for girls. 



The Phase III 
Evaluation Found No 
Significant Effects of 
Exposure to the 
Campaign on Youth 
Drug Use Outcomes 
Other than Limited 
Unfavorable Effects 
on Marijuana 
Initiation 



Westat reported that the NSPY data showed some declines in self-reported 
lifetime and past-month use of marijuana by youth over the period from 
2002 to 2004, and these trends in NSPY were consistent with trends in 
other national surveys of drug use over these years. Westat also reported 
that the NSPY data showed declining trends in youth reports of offers to 
use marijuana. However, Westat cautioned that because trends do not 
account for the relationship between campaign exposure and changes in 
self-reported drug use, drug use trends alone should not be taken as 
definitive evidence that the campaign was responsible for the declines. On 
the basis of the analysis of the relationship between exposure to campaign 
advertisements and youth self-reported drug use in the NSPY data — 
assessments that used statistical methods to adjust for individual 
differences and control for other factors that could explain changes in self- 
reported drug use — for the entire period covered by its evaluation, Westat 
found no significant 23 effects of exposure to the campaign on initiation of 
marijuana by prior nonusing youth. The only significant effect indicated in 
Westat’ s analysis of the relationship between campaign exposure and self- 
reported drug use was an unfavorable effect of exposure on marijuana 
initiation — that is a relationship between campaign exposure and higher 
rates of initiation — for one round of NSPY data and similar unfavorable 
effects of campaign exposure on marijuana initiation among certain 
subgroups of the sample (e.g., Y2 l A- to 13-year-olds and girls). Westat found 
no effects of campaign exposure on rates of quitting or use by prior users 
of marijuana. 



23 In discussing Westat’s findings, any references to significance refers to statistical 
significance. 
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Westat Tracked Trends in 
Marijuana Use from 
Several Sources and 
Reported That the Trend 
Data by Themselves Were 
Insufficient to 
Demonstrate Effects of 
Exposure to the Campaign 



Westat tracked trends in self-reported use of marijuana by youth and 
trends in youth reports of offers to use marijuana for the period from 
2000 to the first half of 2004 to determine if there were significant declines. 
Westat also assessed these trend data for changes occurring since 2002, 
or during the period of the redirected campaign. Westat’ s trend analysis 
was designed to provide supportive but not definitive evidence for 
campaign effects. 

In its trend analysis, Westat compared trends in self-reported drug use — 
lifetime, past year, and past month — in the NSPY with trend data on self- 
reported drug use from three other nationally representative surveys of 
drug use — Monitoring the Future, the Youth Risk Behavior Surveillance 
System (YRBSS), and the National Survey on Drug Use and Health. 24 Both 
MTF and YRBSS are school-based surveys, and NSDUH is a household 
survey that provides estimates of drug use by the civilian, 
noninstitutionalized population of the United States aged 12 years and 
older. Methodological differences between the school-based surveys — 
MTF and YRBSS — and the household surveys — NSPY and NSDUH — have 
been shown to account for the some of the differences in estimates of 
marijuana use. 



According to Westat’ s analysis, the surveys of self-reported marijuana use 
show some similarities and differences in trends depending upon the 
measure, age group, or subperiod covered within the longer 2000 to 2004 
period. For example, the MTF data generally show declines in lifetime, 
past-year, and past-month self-reported drug use for 8th, 10th, and 
12th graders over the years from 2000 to 2004, although only some of the 
year-to-year differences in the MTF self-reported drug use data were 
statistically significant. Nonetheless, for the subperiod from 2002 to 2004, 
MTF data show statistically significant declines in past-year and past- 
month use for 8th graders and past-year use for 10th graders, and the 
NSPY data also show statistically significant declines in past-month use 
from 2002 to 2004 for youth aged 12 x /2 to 18 years old and for 14- to 18-year- 
olds. On the other hand, the MTF data suggest a decline in past-year and 
past-month use by 10th graders from 2000 to 2002, but the NSPY data 
suggest an increase in past-month marijuana use during this period. 25 
Further, the data from NSDUH for 2000 and 2001 also show statistically 



24 This survey was formerly known as the National Household Survey on Drug Abuse. 

25 Westat points out, however that the MTF decline in use among 10th graders between 
2001 and 2002 was within the statistical confidence limits of NSPY. 
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significant increases in lifetime, past-year, and past-month marijuana use 
among youth aged 12 to 17, statistically significant increases in lifetime 
and past-year marijuana use for youth aged 16 to 17, and a statistically 
significant increase in past year use for youth aged 14 to 15. The pattern of 
increase in NSDUH data from 2000 to 2001 is consistent with the 2000 to 
2002 increases in past-month use in NSPY, but they differ from the MTF 
trends over this period. 

All four surveys generally show declines in marijuana use beginning in 
2002, but not all of the declines are statistically significant. Both MTF and 
NSPY show some statistically significant declines since 2002, and while 
NSDUH and YRBSS show declines, the declines were not statistically 
significant. These declines starting in 2002 coincide with the redirected 
campaign and the introduction of the Marijuana Initiative. 

Despite the concurrence of the trend data from all sources for the 2002 to 
2004 period, Westat concluded that the existence of declining trends in 
self-reported drug use by themselves do not provide definitive evidence 
that the campaign caused the declines because factors other than the 
campaign also could affect behavior. For example, changes in high-school 
completion rates among youth could affect drug use behaviors, as high 
school dropouts may have more involvement with drugs than youth who 
stay in school. Additionally, declines in self-reported drug use that began 
before the initiation of phase III of the campaign could not have been 
caused by the campaign. The declines reported in MTF began prior to the 
start of phase III of the campaign; therefore, factors other than the 
campaign had to have been responsible for the start of the decline 
occurring in these data. Further, ONDCP also has acknowledged the 
limitations of trends in the national surveys for determining whether 
changes in drug use were the result of the campaign. ONDCP’s Office of 
Programs, Budget, Research and Evaluation wrote about the MTF, YRBSS, 
and NSDUH: 26 



“They provide policy makers with broad indicators of the success of 

policy. . .However, they will not be able to answer the critical question of whether 

these changes were the result of the Media Campaign. These surveys do not ask 



26 At the time that ONDCP prepared this document, NSDUH was still known as the National 
Household Survey on Drug Abuse, or NHSDA. 
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respondents about their exposure and reactions to the messages of the Media 
Campaign that can then be linked to their drug-related attitudes and behavior.” 2 ' 



Westat Reported That 
Trends in Marijuana Offers 
Declined over Time, but 
Factors Other than the 
Campaign Contributed to 
Changes in Offers 



Westat assessed trends in youth reports of receiving offers of marijuana — 
whether anyone had ever offered youth marijuana and the frequency of 
offers within the past 30 days. Marijuana offers are closely related to 
marijuana use, and the campaign aired messages that encouraged 
resistance to offers of marijuana. Over the 2000 to 2004 period, Westat 
found significant increases in the percentage of youth reporting that they 
had never received offers, and it also found significant decreases in the 
percentage of youth reporting that they had received offers in the prior 
month. Westat also found significant changes in offers over 2002 to 2004, 
during the period of the redirected campaign, and these changes were 
generally consistent with the trends for the overall 2000 to 2004 period. 
Further, on the basis of longitudinal analysis of the relationship between 
offers in one period and marijuana use in the subsequent period among 
youth who were nonusers in an initial survey round — an analysis that 
assesses whether offers precede use or are simply a correlate of it — 
Westat found that youth who reported having received a marijuana offer at 
one period were much more likely — between three and seven times more 
likely, depending upon age group — to have initiated marijuana use at a 
following period than nonusing youth who reported never having received 
such an offer. However, as Westat reported, while the findings on offers 
are favorable, they cannot be ascribed to the campaign because they may 
be caused by other factors, as the analysis of the relationship between 
offers and use did not take into account other factors that could affect use. 



2 ‘Office of Programs, Budget, Research and Evaluation, Office of National Drug Control 
Policy, “Youth Drug Use and the National Youth Anti-Drug Media Campaign,” February, 
2001 (Washington, D.C.: Executive Office of the President), p. 16. 
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On the Basis of Its Analysis 
of the Association between 
Exposure and Drug Use 
Outcomes, Westat Found 
No Evidence That 
Exposure to the Campaign 
Affected Initiation or 
Cessation of Marijuana 
Use 



From its longitudinal analysis of associations between exposure and 
initiation of marijuana use, Westat found no evidence that increased 
exposure to the campaign reduced youth’s initiation of marijuana use. 
Westat’s longitudinal analysis assessed the effects of exposure at one 
survey wave on marijuana initiation at a subsequent survey wave, 
controlling for potential confounding variables that could affect the 
exposure initiation relationship. Westat assessed the effects of two types 
of exposure on initiation of marijuana use — general exposure and specific 
exposure. General exposure represents the sum of recalled exposure to 
anti-marijuana advertising in four types of sources of advertisements — 
television and radio, movies and videos, print media including newspapers 
and magazines, and outdoor media. Specific exposure represents the sum 
of recalled exposure to youth-targeted individual campaign television 
advertisements that had been aired in the 60 days prior to an interview. 

Westat found no significant effects of the level of general exposure on 
marijuana use initiation, either over the entire period of the campaign or 
between subperiods as defined by survey rounds. 28 Westat also found no 
overall effects of levels of specific exposure on marijuana initiation during 
the entire period of the campaign, but it found one significant association 
between specific exposure and marijuana use initiation that occurred in 
the data from wave 7 and its wave 9 follow-up, or during the period of the 
Marijuana Initiative. Wave 7 was the first complete survey wave covering 
exposure to the Marijuana Initiative. The significant association from this 
analysis was that higher levels of specific exposure were associated with 
higher levels of initiation of marijuana use among previously nonusing 
youth. 

Westat also examined the longitudinal relationships between exposure 
and initiation for nine subgroups of youth (two sexes, three race/ethnicity 
groups, two risk groups, and two nonoverlapping age groups). For several 
subgroups, it found significant associations between specific exposure and 
marijuana initiation. These associations were in a direction that was 
unfavorable to the campaign, in that greater specific exposure was 
associated with higher levels of initiation. The subgroups for which these 
unfavorable associations were most pronounced included 12V6- to 13-year- 
olds, girls, African Americans, and lower risk youth. 



28 Westat’s assessed the exposure-initiation relationship using data from survey rounds 
1 and 2, survey rounds 2 and 3, and within survey round 4; it assessed the exposure- 
initiation relationship between waves 6 and 8 and waves 7 and 9. 
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On cessation and reduction of marijuana use, Westat assessed two 
outcomes among current marijuana users: the rate at which they quit using 
marijuana and their frequency of use. The frequency of use measure 
allowed for campaign effects to be observed if users did not quit but 
reduced their use of marijuana. Westat estimated that the quit rate — the 
percentage of prior-year users reporting that they no longer used 
marijuana — among prior-year users of marijuana was 24.8 percent. 
However, it found no statistically significant association between general 
exposure and quitting or between specific exposure and quitting. It also 
found that among adolescent marijuana users, the frequency of use — 
increase, decrease, or no change — was not affected by exposure to the 
campaign. 



Conclusions 



A well-designed and executed multiyear study of the impact of the ONDCP 
anti-drug media campaign on teen initiation of drug use, or cessation of 
drug use, shows disappointing results for the campaign. The study 
provides no evidence that the campaign had a positive effect in relation to 
teen drug use, and shows some indications of a negative impact. Some 
intermediate outcomes, such as parents talking with children about drugs, 
and doing fun activities with their children, showed positive results in that 
the media campaign encouraged parents to adopt these behaviors. 
However, other intermediate outcomes, such as parents’ monitoring of 
their children’s behavior, were not shown to be affected by the campaign. 
Moreover, the evaluation did not provide evidence that intermediate 
outcomes that showed positive results translated into greater resistance to 
drugs among the teenage target population. 

Unfavorable preliminary findings from the evaluation were reported by 
Westat in 2002. Beginning in 2002, ONDCP took a number of steps that 
were intended to strengthen the power of the campaign to achieve positive 
results. These steps included more rigorous ad copy testing and a 
concentration on anti-marijuana messages. However, the post-2002 results 
yielded no evidence of positive impacts and some evidence of negative and 
unintended consequences in relation to marijuana use. Specifically, 
exposure to advertisements during the redirected campaign was 
associated with higher rates of marijuana use initiation among youth who 
were prior nonusers of marijuana. 

Most parents and youth recalled exposure to the campaign messages and, 
further, they recognized the campaign brand. Thus, the failure of the 
campaign to show positive results cannot be attributed to a lack of 
recognition of the messages themselves. This raises concerns about the 
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ability of messages such as these to be able to influence teen drug 
attitudes and behaviors. It raises questions concerning the understanding 
of the factors that are most salient to teens’ decision making about drugs 
and how they can be used to foster anti-drug decisions. 

Westat’s evaluation is centered on this particular configuration of a media 
campaign as it was presented from 1999 to 2004, and its results pertained 
to the campaign nationwide. It cannot be construed to mean that a media 
campaign that is configured differently from this one cannot work. Nor 
should its results be construed to mean that in some locations, for some 
groups of youth, the campaign did not have an effect on drug use. 

However, substantial effort and expertise were brought to the task of 
designing the advertisements from the outset, and the 2002 redirection of 
the campaign placed even greater emphasis on copy testing and enhanced 
ONDCP oversight. This casts some doubt on the notion that a better media 
campaign can lead to positive results. 

It is also important to note that two recent smaller studies in three 
locations have provided evidence of a limited effect of the campaign for 
some youth, and it is quite possible that additional analyses of the NSPY 
data using different methods or measures may find other effects of the 
campaign, at least for some adolescents, than have been produced by 
Westat’ s evaluation team. The data from the evaluation have only recently 
been made available to academic and other researchers, and while the 
analyses undertaken by Westat are, as we have noted elsewhere, 
appropriate and thorough, they are not exhaustive. 

It is heartening that surveys intended to measure teen drug use, such as 
Monitoring the Future, are showing declines in marijuana use in recent 
years. Indeed, NPSY also shows some evidence of a decline in drug use 
among teens. However, Monitoring the Future and other surveys of teens 
concerning drug use are not linked to exposure to the media campaign, 
and NPSY shows no relationship between anti-drug media campaign 
exposure and favorable drug outcomes for teens. This seems to indicate 
that other unidentified factors, other than the anti-drug media campaign, 
are affecting drug use decisions among teens. 

Although ONDCP has pointed to declines in teen drug use and credited the 
campaign along with other prevention efforts as contributing to significant 
success in reducing teen drug use, trend data derived from the Monitoring 
the Future survey that show declines in teen marijuana use from 2001 to 
2005 do not explicitly take into account exposure to the campaign, and 
therefore, by themselves, cannot be used as evidence of effectiveness. 
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ONDCP has indicated in the past, and we concur, that because these 
surveys cannot link their results with the media campaign, they do not 
measure campaign effectiveness. The evaluation of the media campaign 
reinforces the lack of linkage between the media campaign and teen drug 
use behavior. 




It is important to note that virtually all social science research is 
imperfect. Attempting to systematically observe and document human 
behavior in real-world settings is a daunting task given the extremely wide 
variation in both humans and settings. We believe that the evaluation of 
the ONDCP media campaign is credible in that it was well designed given 
the circumstance of the campaign, and appropriately executed. 


Matter for 

Congressional 

Consideration 


In light of the fact that the phase III evaluation of the media campaign 
yielded no evidence of a positive outcome in relation to teen drug use and 
congressional conferees’ indications of their intentions to rely on the 
Westat study, Congress should consider limiting appropriations for the 
National Youth Anti-Drug Media Campaign beginning in the fiscal 2007 
budget year until ONDCP is able to provide credible evidence of the 
effectiveness of exposure to the campaign on youth drug use outcomes or 
provide other credible options for a media campaign approach. In this 
regard we believe that an independent evaluation of the new campaign 
should be considered as a means to help inform both ONDCP and 
Congressional decision making. 


Agency Comments 
and Our Evaluation 


We provided a draft of this report to the Director of the Office of National 
Drug Control Policy for comment on July 31, 2006. ONDCP provided us 
with written technical comments on the report, which we incorporated 
where appropriate. In addition, ONDCP provided written comments about 
our report in which it raised a question about our matter for congressional 
consideration and outlined a number of concerns that it had with our 
report on Westat’s findings. These written comments are reproduced in 
appendix II. In our evaluation of ONDCP’s written comments, we address 
each of the other concerns in the order ONDCP presented them. 


Westat Evaluation’s Role in 
Judging the Impact of the 
Advertising Campaign 


ONDCP comments that Westat’s evaluation is ill suited to judge the impact 
of an advertising campaign in part because Westat attempted to establish a 
causal relationship between exposure and outcomes, and this, ONDCP 
indicates, is something that major marketers rarely attempt because of its 
difficulty. ONDCP writes, “we take issue with the fundamental method 
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pursued by Westat and GAO, and therefore, believe that the study’s 
findings are deeply flawed.” We find this response surprising for a number 
of reasons. First, ONDCP is on record as stating that the evaluation 
conducted by Westat would be the means to assess the impact of the 
campaign. Indeed, in February, 2001, in the ONDCP publication entitled 
Youth Drug Use and the National Youth Anti-Drug Media Campaign, 
ONDCP states: 

“ONDCP, on the other hand, is measuring the impact of the Media Campaign with 
a thorough, rigorous, and independent evaluation. The nationally representative 
evaluation is being conducted for ONDCP by the National Institute on Drug Abuse 
(NIDA). . . .The evaluation is a 4-year longitudinal study of parents’ and their 
children’s exposure and response to the Media Campaign.... ONDCP will be able 
to assess the extent to which changes in anti-drug attitudes and beliefs or drug 
using behavior can be attributed to the Media Campaign.” 29 

ONDCP officials had opportunities during the evaluation to raise concerns 
about Westat’ s design and its efforts to establish a link between exposure 
to the campaign and outcomes, but we are not aware of their having done 
so. However, we are aware of ONDCP’s participation in a NIDA-sponsored 
expert panel review of Westat’ s evaluation that was held in August 2002. 
Our review of the minutes of that meeting reveals that while an ONDCP 
official raised concerns about issues such as assessing the nonadvertising 
components of the campaign and the number of interim reports, ONDCP 
officials did not at that time raise concerns that the evaluation was 
fundamentally flawed. The consensus of the expert panel was that 
Westat’ s evaluation was “pretty impressive” given the challenges presented 
by the absence of baseline data and of an experimental design. Panel 
members also asserted that Westat’s use of propensity score models to 
isolate the effects of the campaign was termed both “sensible” and “state- 
of-the-art.” 

ONDCP further states that major advertisers evaluate the success of their 
campaigns by rigorously testing advertisements prior to airing and by 
developing correlations between messages and consumer attitudes and 
behavior. While we do not dispute whether this is a commonly used 
approach among major advertisers, we believe that in assessing the 



29 0ffice of Programs, Budget, Research and Evaluation, Office of National Drug Control 
Policy, “Youth Drug Use and the National Youth Anti-Drug Media Campaign,” February, 
2001 (Washington, D.C.: Executive Office of the President), p. 3. 
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expenditure of public funds researchers should attempt, where feasible, to 
establish causal relationships or use research designs that attempt to 
isolate the effects of federally funded interventions. While we 
acknowledge that establishing causal relationships is difficult, we maintain 
that Westat used sophisticated and appropriate statistical methods that 
aimed to isolate the effects of recalled exposure to the campaign on youth 
drug use. Further, adopting a methodology that relies upon correlations 
between advertising messages and an outcome, such as reductions in 
youth drug use, without attempting to take into account many of the other 
factors that could affect drug use allows for too many post hoc 
explanations of findings. Westat’s analysis included socioeconomic 
factors, parent characteristics, television viewing habits, risk of using 
drugs, and sensation-seeking tendencies to be able to determine whether 
exposure was related to drug use net of the influences of these factors. We 
conclude, on the basis of our assessment of Westat’s methods, that 
exposure to campaign messages generally did not influence youth drug 
use net of these other influences. 

ONDCP notes that correlational findings have been used to assess anti- 
tobacco advertising campaign results. We have not reviewed the anti- 
tobacco campaign and cannot comment on its relationship to youth 
smoking prevalence. We notice, however, that in ONDCP’s comments on 
“Consequences of Further Budget Cuts,” it appears to contradict its 
statements about establishing causal relationships to determine the effect 
of advertising campaigns. ONDCP writes, “Previous studies have 
established a relationship between exposure to anti-tobacco messages and 
smoking rates among teens.” ONDCP goes on to draw an analogy between 
anti-smoking messages and anti-drug messages to write, “We should 
expect similar results for illicit drug use if anti-drug messages decline.” 
These statements emphasize very directly the same kind of causal 
relationships that ONDCP cites as not appropriate in its opening 
comments. 

We also note that ONDCP indicates in its comments that it has made 
multiple refinements to the media campaign on the basis of earlier findings 
from the Westat study. This seems to be inconsistent with a position of 
major concerns with the fundamental soundness of the study. 

Finally, the three research papers that ONDCP cites on page 2 of its 
comments on “Conflicting Evidence from Other Research” all use 
exposure-response methodologies that are analogous to Westat’ s and all 
attempt to isolate the causal effects of exposure either to ONDCP’s 
campaign or to other media campaigns. Thus, it would seem that ONDCP’s 
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comment that efforts to isolate causal effects of media campaigns are 
fundamentally flawed would also apply to these three studies. 



ONDCP Made Campaign 
Changes as a Result of 
Westat Interim Findings 



ONDCP indicates that it has sought to improve the performance of the 
media campaign by using the results from the Westat study and other data. 
We are aware that ONDCP redirected the campaign in 2002 in response to 
Westat’s interim findings that indicated some negative impacts of the 
campaign on youth marijuana use. However, the 2002 to 2004 Westat study 
results also did not show positive outcomes. Westat’ s study is the only 
national evaluation of the campaign. Although Monitoring the Future 
provides context regarding general drug trends among youth, as ONDCP 
has stated: 



“These surveys [MTF, the National Household Survey on Drug Use, and the Youth 
Risk Behavior Survey] will permit the determination of whether drug use behavior 
and related attitudes and beliefs changed after the launching of Phase III of the 
Media Campaign in mid-1999. However, they will not be able to answer the critical 
question of whether these changes were the result of the Media Campaign. These 
surveys do not ask respondents about their exposure and reactions to the 
messages of the Media Campaign that can then be linked to their dmg-related 
attitudes and behavior.” 30 

More recently in late 2005, ONDCP launched a newly designed campaign. 
The impact of this campaign is not known and should be independently 
evaluated. 



Other Youth Drug Use 
Findings 



ONDCP believes we did not provide adequate discussion of studies that 
report findings contrary to those of Westat. Our report mentions two of 
the three studies that ONDCP identifies — the Longshore and Palmgreen 
studies. Our report does not mention the third study, Slater, because it 
focused on a different anti-drug media campaign approach and not on the 
ONDCP media campaign. Overall, these studies’ findings are not 
necessarily “contrary” to Westat’ s findings. Rather, they assess small slices 
of the youth population or particular circumstances (such as other 
programs that could reinforce an anti-drug message) and find some 
positive results. The Westat national findings do not preclude the findings 
of positive results for some subpopulations of youth. The Palmgreen study, 



“Office of Programs, Budget, Research and Evaluation, Office of National Drug Control 
Policy, “Youth Drug Use and the National Youth Anti-Drug Media Campaign,” February, 
2001 (Washington, D.C.: Executive Office of the President), p. 16. 
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for example found a positive effect for the media campaign on high- 
sensation-seeking youth, but did not find an effect on non-high-sensation- 
seeking youth in the two moderate size communities in which the study 
was conducted. The distribution of these youth in the nationwide 
population could be consistent with both studies being correct. Our 
objective was to assess the Westat study as a national evaluation of the 
impact of the national campaign. In the Slater study, after being trained in 
the use of campaign media materials, leaders in each of eight communities 
that received a media campaign were allowed to develop their own media 
strategies and were able to use whatever materials they chose or 
developed on their own. This approach emphasized the flexibility to adopt 
different media strategies deemed appropriate by individual communities 
and not the use of a single national strategy. 

ONDCP expressed concern that we had not discussed Westat’s hypothesis 
concerning why the campaign might have contributed to youth 
experimentation with marijuana. We are unable to draw a conclusion 
about this hypothesis based on Westat’s report, nor do we have additional 
information upon which to base an assessment. ONDCP also faults our 
report for not discussing other potential competing explanations for the 
substantial downturn in teen drug use and increase in anti-drug attitudes. 
Although this is beyond the objectives of this report, we note that multiple 
other indicators of youth responsibility also seem to be trending in a 
positive direction at the same time that MTF reports declines in youth drug 
use. For example, from 1991 through 1999, the teen pregnancy rate 
declined by 27 percent and from 1991 through 2002, the teen birth rate fell 
30 percent. Similarly, the number of juvenile homicides declined by 
44 percent from 1993 to 2002, and the juvenile violent crime arrest rate fell 
by more than 40 percent from 1994 to 2003. All of these trends — including 
declines in drug use — could be related to broader environmental, familial, 
or other influences. The coincidence of these trends with drug use trends 
indicates that factors other than the campaign could be responsible for the 
decline in drug use and points to the necessity of trying to isolate the 
effects of the campaign, rather than relying upon simple correlations. 



Steps Taken to Remedy ONDCP states that it has taken extensive “due diligence” steps that are 
Potential Problems briefly acknowledged in our report, but that our report “fails to 

acknowledge the thoroughness of our actions to identify, assess, and 
attenuate any possible negative consequences of the campaign once 
Westat reported the possibility of such an effect.” Apart from those actions 
described in Westat’s evaluation reports, a full discussion of the steps that 
ONDCP took in response to Westat’s interim evaluation reports that 
highlighted the possibility of unintended negative consequences of 
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ONDCP Cites Major 
Changes in Campaign 



ONDCP Offers an 
Alternative Explanation 
for Counterintuitive 
Results 



exposure to the campaign on youth initiation of marijuana was not salient 
to our assessing whether Westat took appropriate steps to address the 
evaluation implementation challenges that it faced. However, Westat’s 
findings for the period from 2002 to 2004 showed that the campaign also 
was not effective after ONDCP took these steps. 



ONDCP states that the campaign is substantially different from what it 
was when the last data were collected by Westat more than 2 years ago. 

We are not in a position to comment on ONDCP’s new campaign (“Above 
the Influence”), launched in November 2005, as these current efforts are 
beyond the scope of our report and outside the time frame of the Westat 
data collection. At this time, neither we nor ONDCP have empirical 
information with which to assess this revised campaign. However, 

Westat’ s evaluation showed that neither the campaign as initially 
implemented nor the redirected campaign implemented after 2002 was 
effective. Hence, although a new and improved campaign may be effective, 
Westat’ s findings raise concerns about whether any campaign can affect 
youth drug use, especially since the lack of effect does not seem to be 
related to recognition of campaign ads, but rather to subsequent impact on 
attitudes and behaviors. Finally, ONDCP cites the receipt of awards from 
both the advertising and communications industry for its newest 
campaign. While laudable, these awards are not evidence that the new 
campaign will change youth drug attitudes and behavior. Only an 
independent evaluation can assess the current campaign’s effectiveness. 



ONDCP stated that there is growing research evidence showing that 
asking people a question about their future behavior influences the 
subsequent performance of the behavior in question. ONDCP then 
indicates that the use of a panel design for the Westat study with repeated 
interviews of youth concerning drug attitudes and behaviors might, itself, 
have resulted in increased perceptions that drug use is widely pervasive 
among youth. If, during the course of the Westat study, ONDCP and NIDA, 
who acted as monitor for the study, felt that the study itself — that repeated 
interviews of youth by Westat concerning the campaign and drug attitudes 
and behavior — was resulting in a negative effect, it would have been 
appropriate for them to discontinue the study to avoid potential harm to 
subjects. Although ONDCP raised this issue in its comments to us, neither 
ONDCP nor NIDA mentioned this issue in any of our previous meetings 
specific to this engagement. 
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ONDCP Takes Issue with ONDCP said that the “long delay” in receiving our assessment of the 
the Timing of Our Review Westat report has prevented it from making progress on the next round of 

evaluation. We note that Westat’s draft final report was not made available 
to us until spring 2005 (not 2 years ago as seems to be indicated in 
ONDCP’s comments). The volume of reports from the 4 '/Ay ear study, and 
the complexity of the review required a great deal of time from our most 
skilled social scientists and statisticians. Time was required to ensure that 
our review of the Westat study was both comprehensive and correct. 



Points Concerning Our 
Matter for Congressional 
Consideration 



ONDCP said that our matter for congressional consideration — that 
Congress consider limiting appropriations until ONDCP is able to provide 
credible evidence of the effectiveness of exposure to the campaign on 
youth drug use outcomes — offers insufficient detail concerning how to 
demonstrate satisfactory evidence of progress and that it was puzzled by 
our lack of recommendations to ONDCP for improving the campaign. Our 
mandate was to assess Westat’ s evaluation and to draw conclusions about 
the reliability of its findings so that Congress could make decisions about 
funding for the campaign, and developing suggestions for improvements to 
the media campaign itself was beyond our scope. In so doing, we focused 
on Westat’ s methods and efforts to address challenges in implementing the 
evaluation. Our matter for congressional consideration was intended to 
allow ONDCP to explore a number of approaches to providing credible 
evidence of campaign effectiveness to Congress. Our report clearly 
indicates that one approach is the one applied in the Westat evaluation, 
which is the focus of this report, but we do not want to rule out other 
approaches. At the same time, we acknowledge that providing such 
evidence is not easy. 



ONDCP Posits 
Consequences of Further 
Budget Cuts 



ONDCP states that further budget cuts to the campaign could have far- 
reaching and unfavorable consequences in youth drug use. Given that the 
Westat findings show that the campaign was not having a positive impact, 
we found no evidence that a reduction in campaign advertisements would 
have a negative impact. ONDCP cites the 2005 MTF as an indicator of 
media campaign effectiveness by indicating that the reduction in anti-drug 
messages has resulted in a flattening of 8th graders’ perception of risk. 
Again, as ONDCP has indicated, the relationship cannot be assessed with 
MTF because it does not ask respondents about their exposure and 
reactions to the messages of the media campaign that can then be linked 
to their drug-related attitudes and behaviors. 
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Failure to continue the media campaign’s efforts, according to ONDCP, is 
“raising a white flag to those who favor drug legalization, with the 
expectation that youth drug use soon would begin to rise, reversing years 
of hard-earned positive news.” In our view, on the other hand, 
continuation of programs that have been demonstrated not to work diverts 
scarce resources from programs that may be more effective. 

We are sending copies of this report to other interested congressional 
committees and the Director of the Office of National Drug Control Policy. 
We will make copies of the report available to others upon request. In 
addition, the report will be available at no charge on GAO’s Web site at 
http://www.gao.gov. 

If you or your staff have any questions about this report, please contact 
either Nancy Kingsbury at 202-512-2700 or by e-mail at 
KingsburyN@gao.gov or Laurie Ekstrand at 202-512-8777 or by e-mail at 
EkstrandL@gao.gov. Contact points from our Offices of Congressional 
Relations and Public Affairs may be found on the last page of this report. 
Key contributors are listed in appendix III. 



Sincerely, 




Nancy Kingsbury, Managing Director 
Applied Research and Methodology 






Laurie E. Ekstrand, Director 
Homeland Security and Justice 
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Appendix I: Westat’s Methods for Addressing 
Evaluation Implementation Issues 



This appendix provides additional details about how Westat’s addressed 
evaluation implementation issues related to the coverage of the National 
Survey of Parents and Youth (NSPY), sample attrition, and its analytic 
methods. 



Coverage in the NSPY The NSPY was a nationwide household survey of youth aged 9 to 18 and 

their parents. Westat used a dual-frame sampling frame — or list of the 
members of the population from which the sample was ultimately 
selected. One frame — the area frame — consisted of housing units that had 
been built by late 1991; the second frame — the building permit frame — 
consisted of building permits issued between January 1990 and December 
1998 for new housing. 1 Combined, these frames constituted an estimated 
98 percent of dwelling units nationwide that existed by the end of 1998. 

A household had to meet two criteria in order to be eligible to be included 
in the NSPY sample: It had to (1) contain children within a specified age 
group and (2) be a housing unit that was built before April 1, 1990, was a 
mobile home, or was selected from a roster of building permits for new 
housing units issued between January 1990 and December 1998. To 
identify households that met these conditions, Westat drew a sample of 
dwelling units and from this sample it screened households to determine 
their eligibility for inclusion in the NSPY, that is, whether a household 
contained children in a specified age group, where the specified age 
groups were children aged 9 through 13, 12 and 13, or 9 through 18. 

According to estimates provided by Westat, after completing enrollment in 
the NSPY — which occurred during waves 1 through 3 — the NSPY sample 
covered more than an estimated 95 percent of occupied dwelling units 
(households) nationwide. From its sample of occupied dwelling units, 
Westat developed rosters of households that were believed to contain 
youth in the target age range. At this second stage of sample enumeration, 
Westat experienced a drop-off in the coverage of households that were 



housing units built after 1998 had no chance of selection in either sampling frame. Also, a 
housing unit had no chance of selection if it had been built during the 1990s in a 
jurisdiction where no permit was required. Finally, modular housing built during the 1990s 
was inadvertently omitted from the permit sample. Any biases resulting from excluding 
housing units built after 1999 are likely to be small, as they constituted a small fraction of 
all housing units in the NSPY sampling frame, and they were accounted for by Westat’s 
poststratification adjustments. For example, housing units built after April 1999 accounted 
for an estimated 1.0 percent of all housing units in existence in the time period covered by 
the wave 1 sample. 
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believed to be eligible for inclusion in the sample. The number of eligible 
households enumerated in the NSPY was 30 percent smaller than the 
number expected from the 1999 Current Population Survey (CPS) data. 

According to Westat, coverage losses in the NSPY could have occurred for 
several reasons: (1) because an interviewer may have decided to classify a 
household as an ineligible household rather than as a nonresponding 
household, (2) because the household respondent took cues from the 
screening questions to avoid selection into the sample by giving an 
incorrect answer, or (3) because the doorstep enumeration process was 
considered to be intrusive. Westat reported that it could not conclusively 
rule out the first explanation for coverage losses. However, it undertook 
sample validation procedures that examined whether ineligible 
households in the recruitment waves were misclassified, and it found 
none. Neither Westat nor the National Institute on Drug Abuse reported 
that undercoverage was primarily due to respondents avoiding selection 
into the sample by taking cues from the screening questions and giving 
incorrect answers as a way to avoid selection into the sample. Overall, 
Westat reported that the main reason for undercoverage was the rostering 
component of the survey, which required actual entry into the home, and 
led to “a great many respondents” asking the interviewer to come back at a 
later date, only to repeat the request when the interviewer reappeared. 
Westat inferred that this represented passive refusal to participate. 
Therefore, according to Westat, most of the coverage losses occurred 
during the doorstep screening process in which simple, focused screening 
questions about the composition of the household were used to identify 
households from which to sample eligible youth. 



NSPY and CPS 
Comparisons of 
Distributions on Analyzed 
Variables 



In response to questions from us, Westat provided data that indicated that 
the coverage losses in the NSPY did not result in differences in the 
estimated distributions of population characteristics from the NSPY as 
compared with those estimated from the CPS data. In other words, the 
distributions of characteristics of eligible households with youth included 
in the NSPY were broadly consistent with a variety of corresponding 
distributions from the 1999 Current Population Survey. 



The comparisons of NSPY-estimated populations to CPS-estimated 
populations were based on weighted NPSY estimates, where the weights 
adjusted for nonresponse at the doorstep and household enumeration 
(roster) stages, and the weights also reflected the differential probabilities 
of retaining a household for the NSPY depending on the screener group to 
which it was applied. These weights were calculated prior to Westat’s 
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poststratification calibration techniques, which brought the estimated 
NSPY population totals into line with the estimated CPS population totals. 
Hence, if upon using the weights based only on the probability of selection 
and nonresponse adjustments, the population characteristics in the NSPY 
differed widely from those derived from the CPS, this would constitute 
evidence of potential bias in the NSPY sample due to undercoverage. 

Westat compared NSPY and CPS distributions for each of the three 
enrollment waves of the NSPY (waves 1 through 3) on several variables, 
including the race/ethnicity of the householder and the presence of males 
28 years of age or older, the distribution of eligible households by the age 
of the youth in the household, the age and gender distributions of youth, 
and the age distributions of youth by race and ethnicity. Each of these 
comparisons involved discrete subgroups within the focused 
subpopulation of the NSPY. The largest differences between the NSPY and 
CPS estimates arose in the comparison of the distributions by 
race/ethnicity of household and the presence of a male 28 years of age or 
older in the household. Some of these differences could also arise from 
sampling variance, as both the NSPY and CPS estimates are based on 
samples that are subject to sampling errors. Although Westat did not 
provide sampling errors with the estimates that it provided to us, some of 
the differences in distributions could be apparent, as opposed to real, 
differences, in statistical terms. 



Undercoverage in the 
NSPY and Other Widely 
Known and Used 
Longitudinal Surveys 



Coverage issues are not an uncommon problem with surveys that focus on 
relatively small subpopulations within a larger population, such as 
occurred with the NSPY’s focus on youth aged 9 to 18. The NSPY’s target 
population of households with youth aged 9 to 18 focused on a 
subpopulation that, according to 1999 CPS data, constituted about 
25 percent of the roughly 104 million households in the United States. 

The estimated extent of undercoverage of eligible youth in the NSPY was 
comparable to the extent of undercoverage in other well-known and 
widely used longitudinal surveys. Both the National Longitudinal Survey of 
Youth (NLSY) — sponsored by the Bureau of Labor Statistics — and the 
National Immunization Survey of Children (NIS) — sponsored by the 
National Immunization Program (NIP) and conducted jointly by the NIP 
and the National Center for Health Statistics of the Centers for Disease 
Control and Prevention — focus on specific subpopulations, and both 
experienced undercoverage that was comparable to that of the NSPY. The 
1979 NLSY is a nationally representative sample of men and women bom 
in the years 1957 to 1964 who were ages 14 to 22 when first interviewed in 
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1979. It had a coverage rate of 68 percent. The 1979 NLSY has been widely 
used and cited to examine a wide variety of policy issues. As documented 
in the National Longitudinal Surveys’ annotated bibliography, about 
3,100 journal articles, working papers, monographs, and other research 
documents have been catalogued as having used the 1979 NLSY data. The 
target population for the NIS is children between the ages of 19 and 
35 months living in the United States at the time of the interview, and it 
has been conducted annually since 1994. The survey involves the selection 
of a quarterly probability sample of telephone numbers, and the coverage 
has been about 20 percent lower than estimated by two other benchmark 
surveys. Survey data are used primarily to monitor immunization coverage 
in the preschool population in the nation and to provide national, state, 
and selected urban area estimates of vaccination coverage rates for these 
children. 



Sample Attrition across In the NSPY, respondents initially recruited into the sample were to be 
NSPY Interview Rounds tracked for three additional survey rounds that covered about a 3-year 

period following the recruitment round. By the final survey round of the 
NSPY, the cumulative response rate — the percentage of youth or parents 
in eligible households that completed all four interviews — reached 
between 50 percent and 55 percent. These cumulative response rates after 
four survey rounds were determined largely by the response rates during 
the enrollment waves, as postenrollment, Westat was able to track, 
contact, determine eligibility for reinterview, and complete interviews for 
between 82 percent and 94 percent of previously interviewed respondents 
between two successive interview waves. The response rates achieved for 
the first three survey waves — the enrollment waves — were generally 
similar. Specifically, about 74 percent to 75 percent of the dwelling units 
determined to be eligible for the survey in waves 1 through 3 completed 
the household enumeration (or rostering of youth). After obtaining 
consent to conduct interviews from parents and youth, interviewers 
completed extended interviews — that is, completed the full NSPY 
questionnaire — with about 91 percent of the sampled youth in each of 
waves 1 through 3. Among sampled parents, about 88 percent gave 
consent and completed extended interviews in the enrollment waves. 

(See table 2.) 
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Table 2: NSPY Survey Rounds and Response Rates, Sampled and Surveyed Youth 





Rounds and stages of sampling 




Survey waves 




Round 1: enrollment waves 


Wave 

1 


Wave 

2 


Wave 

3 


• 


Percentage of sampled dwelling units for which eligibility was determined 


95.1% 


95.7% 


95.5% 


• 


Percentage of eligible dwelling units completing household roster 


74.4% 


74.6% 


75.3% 


• 


Percentage of youth completing interview 


90.3% 


91 .9% 


91.2% 


• 


Cumulative (overall) response rate, enrollment waves 


63.8% 


65.5% 


65.5% 




Round 2: first follow-up 


Wave 4 




Wave 5 


• 


Percentage of dwelling units (from prior wave) refielded for follow-up 


94.2% 




92.9% 


• 


Percentage of refielded dwelling units for which eligibility was determined 


86.8% 




93.8% 


• 


Percentage of youth completing interview 


93.5% 




93.6% 


• 


Cumulative (overall) response rate 


54.1% 




58.4% 


• 


Follow-up (conditional) longitudinal response rate 


82.2% 




88.8% 




Round 3: second follow-up 


Wave 6 




Wave 7 


• 


Percentage of dwelling units (from prior wave) refielded for follow-up 


85.1% 




89.8% 


• 


Percentage of refielded dwelling units for which eligibility was determined 


93.1% 




92.8% 


• 


Percentage of youth completing interview 


94.7% 




93.8% 


• 


Cumulative (overall) response rate 


53.1% 




56.0% 


• 


Follow-up (conditional) longitudinal response rate 


93.4% 




91.6% 




Round 4: third follow-up 


Wave 8 




Wave 9 


• 


Percentage of dwelling units (from prior wave) refielded for follow-up 


78.7% 




83.5% 


• 


Percentage of refielded dwelling units for which eligibility was determined 


95.9% 




94.8% 


• 


Percentage of youth completing interview 


94.0% 




94.3% 


• 


Cumulative (overall) response rate 


50.2% 




53.4% 


• 


Follow-up (conditional) longitudinal response rate 


92.4% 




93.4% 


Source: Westat, June 2005. 



Across the three follow-up rounds of the NSPY, Westat achieved between 
an 82 percent and a 94 percent longitudinal response rate. Follow-up 
required that respondents be tracked over time and across places, as 
persons enrolled in the sample could move, and their eligibility for a 
follow-up interview had to be determined. For example, youth who turned 
19 years of age between survey rounds would no longer be eligible for 
reinterview, as they were beyond the target age of the campaign. Efforts to 
track individuals prior to the second survey round included verifying 
address change information with the U.S. Postal Service and obtaining 
location information from a national database company. Westat obtained 
updated location information from these sources, and telephone 
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interviewers placed calls to these households to verify the identity of 
respondents. According to Westat, a high proportion of the households 
that moved were contacted and respondents verified their new addresses. 
During the third and fourth survey rounds, Westat used procedures to 
track and verify addresses that were similar to those used to track 
respondents from the first to second survey rounds, although Westat 
modified these procedures as necessary. The key eligibility requirement 
for youth for a follow-up interview was the youth had to be 18 years of age 
or younger at the time of the interview. 

For the first follow-up round — waves 4 and 5 — Westat located individuals 
and determined eligibility for 92 percent of the youth and 92 percent of the 
parents who completed an initial interview during the first round of the 
survey — that is, in waves 1, 2, and 3, and of these youth who were still 
eligible, 94 percent completed an interview. Among parents from the first 
round who were tracked and determined to be eligible in the second 
round, 92 percent completed a second round interview. In the third and 
fourth survey rounds of the NSPY, between 96 percent and 97 percent of 
the youth and parents who had completed prior round surveys were 
tracked and determined to be eligible, and of these, the youth response 
rates were 96 percent and the parent rates were 95 percent. 



Comparisons of 
Respondents and 
Nonrespondents across 
NSPY Survey Waves 



Even with the relatively high follow-up response rates that Westat 
achieved, it is possible that respondents could differ from nonrespondents 
in follow-up rounds, and if so, the NSPY estimates of the effects of 
exposure on outcomes would be biased. Westat provided data that 
compared nonrespondents to the respondents across the three enrollment 
waves, indicating that with some differences, nonrespondents were 
generally similar to respondents with respect to characteristics that might 
affect survey outcomes. Nonrespondents were compared to respondents 
on gender, age at interview, whether both parents were in the household, 
the number of youth in the household, the type of household dwelling, and 
the type of area in which the household was located. For example, apart 
from the three differences below, nonrespondents and respondents were 
similar in characteristics across survey waves: In the three enrollment 
waves, nonrespondents were proportionately older youth than 
respondents; in waves 2 and 3, there were proportionately more youth 
living in cities among nonrespondents than respondents; and in wave 1, 
there were proportionately more youth in the building permit sample 
among nonrespondents than respondents. 
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Differences in Sampling 
Methodologies between 
NSPY and MTF 



Westat compared estimates of drug-use prevalence from the NSPY data 
with those obtained from other national surveys such as Monitoring the 
Future (MTF). While the NSPY estimates of marijuana use prevalence 
differ over some periods covered by the NSPY from those derived from the 
MTF survey of youth in school, differences between the two surveys’ 
sampling frames and methodologies mean that direct comparisons 
between the two surveys must be made with caution and must take the 
methodological differences into account. Specifically, MTF showed a 
decline in marijuana use for some teenage groups during the 2000 to 2002 
period, while the NSPY showed the increases reported above. However, 
the difference in drug use rates reported from the two surveys could 
plausibly arise from differences in the sampling frames. The MTF sampling 
frame covers only youth who are in school and not those who drop out of 
school, who are truant on the survey day, or who are 17- and 18-year-olds 
who have graduated from high school. To the extent that high school 
dropouts and truants have more involvement with drugs than those who 
stay in school, the MTF estimates of drug use may underrepresent drug 
use among all youth of high school age. By comparison, the NSPY 
household survey includes youth who are not enrolled in school in its 
sampling frame. To the extent that dropping out of high school is 
correlated with drug use, and given that dropouts are excluded from the 
MTF sampling frame, differences in drug use between MTF and NSPY 
could reflect the fact that youth enrolled in high school reported drug use 
at different rates from all youth in the general population covered by the 
NSPY, which would include dropouts who may be at higher risk of using 
drugs. 



The Capacity Of the NSPY One challenge in designing surveys to evaluate changes in outcomes as the 
to Detect Reasonably result of an intervention lies in selecting a sample with sufficient power to 

Small Effects detect differences between groups — including the same individuals at two 

points in time — or significant associations among variables, such as 
between levels of exposure to the campaign and outcomes. Sample size is 
a major factor determining a study’s power to detect differences, and 
while larger sample sizes will generally allow researchers to detect smaller 
differences over time, as the size and power of a sample to detect changes 
increases, so too generally does its cost. 

In consultation with the National Institute on Drug Abuse (NIDA), Westat 
chose to compute power for analyses of annual change in a prevalence 
statistic — that is, change in the percentage of a population that reported an 
outcome. For purposes of its power analysis, Westat chose to assume 
different baseline prevalences for parents and for youth of all ages and to 
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assume that the study should be able to detect reliably declines of 
specified sizes. For example, for youth of all ages, Westat assumed a 
baseline prevalence of 10 percent and determined the power of its sample 
for detecting a minimum downswing in an outcome — such as past-month 
drug use — of 2.3 percentage points over a year. 2 The power of the sample 
to detect this difference was well within conventional power criteria. 3 

As reported above, the sizes of differences that Westat’ s sample could 
detect were consistent with the Office of National Drug Control Policy’s 
(ONDCP) goals for the campaign. In early meetings on the design of the 
evaluation of the media campaign, ONDCP officials reported that ONDCP 
had a specific Performance Measures of Effectiveness (PME) system and 
that the campaign was embodied within the first goal of the National Drug 
Strategy, which was to “educate and enable America’s youth to reject 
illegal drugs as well as the use of alcohol and tobacco.” Under this goal, 
ONDCP’s PME proposed targets for reducing the prevalence of past-month 
use of illicit drugs and alcohol among youth from a 1996 base year: by 
2002, reduce this prevalence by 20 percent, and by 2007, reduce it by 
60 percent. ONDCP officials further identified specific targets for the 
media campaign, again with respect to a base year of 1996: by 2002, 
increase to 80 the percentage of youth who perceive that regular use of 
illicit drugs, alcohol, and tobacco is harmful; and by 2002, increase to 
95 the percentage of youth who disapprove of illicit drug, alcohol, and 
tobacco use. To achieve a goal of 80 percent of youth who perceive that 
regular use of marijuana is harmful would require increasing the 1996 
baseline percentage of youth perceiving marijuana as harmful from 
60 percent, as measured by MTF, or by about 3.3 percentage points per 
year from 1996 to 2002. Westat’s sample had sufficient power to detect this 
amount of annual change in youth attitudes. 

The power of the NSPY to detect changes in outcomes due to exposure to 
the campaign also presumes that it was possible to accurately measure 
and characterize exposure to the campaign by the reported number of 
advertisements recalled by respondents. While the general question of 



2 The power to detect differences for upswings in prevalence would depend upon the 
baseline level. However, the power to detect an upswing from a baseline of 90 percent of 
youth would be exactly the same as that for detecting a downswing from a 10 percent 
baseline. 

Specifically, the minimum detectable difference for wave-to-wave changes was at least 
80 percent using a one-sided hypothesis test at the 0.05 level. 
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how exposure to advertisements affected respondents was beyond the 
scope of the evaluation, if by exposure is meant a recognition-based 
task — or encoded exposure — then the NSPY measures of exposure can be 
viewed as valid. According to communications researchers, often what is 
of interest to campaign planners and evaluators is whether the 
presentation of campaign content generates at least a memory trace in 
individuals. At this point, a potential audience member can be said to have 
engaged the campaign’s presentation in a meaningful sense, and this is 
what is meant by encoded exposure. To measure exposure to the 
campaign for both youth and parents, NSPY interviewers asked 
respondents about their recall of specific current or very recent television 
and radio advertisements. 4 

There was variation in recall of advertisements by both youth and parent 
respondents, and this type of variation is needed in order to examine 
associations between levels of exposure and outcomes. For example, for 
the entire campaign, youth reported a median of 12 exposures per month, 
and 76.7 percent reported 4 or more exposures per month. Comparatively 
few youth — about 6 percent — reported less than 1 exposure per month. 
Youth recall of specific exposure also varied, as 41.2 percent of youth 
reported 12 or more television exposures per month throughout the 
campaign while reporting a median of 4.4 exposures to television 
advertisements. Additionally, Westat’ s measures of exposure and 
outcomes have demonstrated sensitivity to detect favorable campaign 
effects among parents. 

Westat’ s test for associations between exposure and outcomes — the 
gamma coefficient — was an ordinal test statistic for whether two variables 
(e.g., exposure and marijuana use initiation) have a montonic, but not 
necessarily a linear, relationship. Therefore, were there nonlinear 
relationships, its test would have allowed for them. Finally, nonrandom 
measurement error in the measure of exposure is unlikely to have biased 
estimates of campaign effects, as if the nonrandom measurement error 
were constant, it would not affect measures of association, and if it was 
not constant, it would be addressed by Westat’ s statistical methods. 



4 Each respondent was presented ads that had been broadcast nationally in the 2 calendar 
months prior to the interview. 
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Westat measured a variety of outcomes for youth and parents and took 
steps to ensure that the measures were consistent with existing research. 
The youth questionnaires included numerous questions that were designed 
to measure exposure to the campaign advertisements and other anti-drug 
messages. The youth question domains included exposure propensity to 
media; current and past use of tobacco, alcohol, marijuana, inhalants, and 
Ecstasy; past discussions with and communication of anti-drug messages 
from parents and friends; expectations of others about respondent’s drug 
use; knowledge and beliefs about the positive and negative consequences 
of drug use; exposure to campaign messages; family and peer factors; 
personal factors; and demographic information. Westat used two separate 
questionnaires for youth of different ages; one questionnaire was used for 
children (aged 9 to 11) and another one was used for teens (aged 12 to 18). 

The NSPY parent questionnaire also included numerous questions that 
were intended to measure parents’ exposure to the campaign’s messages 
and other anti-drug messages. The question domains for parents included 
media consumption; past discussions with child about drug attitudes and 
avoidance strategies; past child monitoring behaviors; self-efficacy of 
discussing drugs with child and monitoring of child’s actions; belief that 
the child is at risk of drug use; belief that drug use has bad consequences; 
exposure to the campaign’s advertising, including brand recognition; 
parent’s own current and past use of tobacco, alcohol, and drugs; and 
demographic information. 

Westat followed generally accepted procedures in developing the survey 
instruments for the NSPY by using information from a prototype prepared 
by NIDA and using information from other surveys that addressed youth 
drug use and prevention. Prior to the phase III evaluation, and in 
preparation for the NSPY, NIDA convened an expert panel to assist in the 
development of the youth and parent questionnaires. The panel, which 
consisted of experts in adolescent drug use prevention and parenting 
behaviors, drafted NSPY survey questionnaires for children, teens, and 
parents, and NIDA shared these prototypes with Westat at the beginning of 
Westat’s evaluation contract. In developing the final questionnaire for the 
NSPY, Westat created a questionnaire development team consisting of 
evaluation experts. In developing the final NSPY questionnaires, the 
Westat team reviewed NIDA’s prototype and other surveys. 
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Westat measured youth drug use by self-reported data on use. We have 
previously cautioned about limitations associated with self-reported data 
on youth drug use . 5 Additionally, the National Research Council (NRC) of 
the National Academy of Sciences also has pointed out limitations 
associated with self-reported drug use in national surveys such as the 
National Survey of Drug Use and Health (NSDUH) and MTF . 6 As NRC has 
pointed out, while self-reported data on drug use may have limitations for 
estimating the actual levels of use at a particular point in time, they may 
not suffer from these same limitations when they are used to assess 
changes in use over time, unless there is reason to believe that attitudes 
about drug use change in ways that affect respondents’ willingness to 
honestly report drug use, or stigma. 

Specifically, if there is a stigma associated with self-reporting drug use, 
that stigma may affect the levels of use reported, as some have argued that 
the propensity of respondents to give valid responses may be affected by 
social pressures. In particular, the incentive to give false negative reports 
may increase over time if drug use becomes increasingly perceived as 
harmful or socially unacceptable. Using data from NSDUH and MTF, NRC 
showed an inverse relationship between the percentages of respondents 
who either disapproved of illegal drug consumption or perceived it to be 
harmful. Thus, as stigma increased, self-reported drug use decreased. As 
NRC cautioned, one could interpret this relationship as indicating that 
changes in stigma are associated with changes in invalid reporting, or as 
stigma increases, false negative reports increase, rather than necessarily 
indicating that as stigma increases, drug use decreases. 

The NRC analysis leads to two inferences: First, if social stigma remains 
constant over time, changes in the propensity to give valid responses 
would be unaffected and estimates of change in self-reported drug use 
would not be biased by social stigma. For the evaluation results, this 
would imply that its measures of changes in self-reported drug use would 
provide valid measures of changes in use, so long as factors other than 
stigma did not affect the propensity to self-report use. Second, if the social 
stigma associated with reporting drug use is inversely related to 
disapproval of illicit drug use or increased perceptions that it is harmful, 



5 GAO: Drug Use Measure?nent: Strengths, Limitations, and Recommendations for 
Improvement, GAO/PEMD-93-18 (Washington, D.C.: June 25, 1993). 

^National Research Council, Informing America’s Policy on Illegal Drugs: What We Don’t 
Know Keeps Hurting Us. National Academy Press, Washington, D.C.: 2001. 
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then the estimates of self-reported drug use are likely to decrease as a 
result of the stigma. According to results from the evaluation, trends in 
youth attitudes and beliefs about illicit drugs changed significantly over 
the entire campaign in a direction that was favorable to the campaign. 
Specifically, the trends in youth attitudes and beliefs about illicit drug use 
meant that youth were more likely to believe, as the campaign went on, 
that use of illicit drugs was likely to have negative consequences. 
Alternatively, the social stigma associated with drug use increased over 
time. If the relationship between stigma and reporting that NRC found 
held and applied to the data in the evaluation of the campaign, this would 
imply that the increased stigma associated with drug use would lead to 
decreases in self-reports of drug use over time. 



Westat’s Analytic 
Methods 



To control for the many factors that could have influenced both exposure 
and outcomes independently of, or in conjunction with, the campaign, 
Westat used propensity scoring methods to match individuals based on 
numerous measured attributes and to create groups of individuals who 
differed on their underlying propensity to be exposed to different levels of 
campaign advertisements. A propensity score is a weighted sum of the 
individual effects of variables in a model that predicts the likelihood of 
exposure to campaign messages. Westat’ s propensity scoring methods 
resulted in the creation of groups of individuals who were statistically 
similar on exposure propensities. These groups can be considered as 
statistical analogues to randomly assigning individuals to different levels 
of exposure. After creating these groups, Westat then analyzed outcomes 
between the groups having different propensities to be exposed to 
campaign messages. 

Westat used ordinal logit models to estimate the chances of being 
exposed, where exposure was measured alternatively as a three- or four- 
level variable — e.g., low, medium, or high exposure . 7 Westat used a myriad 
of variables to predict exposure levels in both the youth and parent 
models. For example, in the youth models, the propensity score models 
included measures of demographic attributes, educational attainment and 
educational aspiration, family and parent background, parent consumption 



Propensity score methods have been demonstrated to be robust against bias associated 
with the specification of incorrect functional forms — e.g., linear rather than quadratic — 
of variables. 
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of television and other media, income and employment, reading habits, 
Internet usage, location of residence in urban areas, among other 
variables. After estimating models, Westat also assessed the balance of 
variables in its propensity models. For propensity models to remove the 
effects of confounding variables from the association between exposure 
and response, it is necessary that the population means of the confounder 
variables not vary across exposure levels. If a confounder is successfully 
balanced, then it will have the same theoretical effect across all exposure 
levels. 

The net result of the propensity scoring models is to provide each 
individual with a score that reflects the individual’s propensity to recall 
advertisements based upon a weighted sum of all of the variables in the 
model. Therefore, while two individuals may differ on the likelihood that a 
particular variable affects their chances of being exposed to messages or 
on their levels of a certain variable — such as age or education — they could 
be similar in their overall propensity to be exposed to campaign messages 
if the differential effects of any individual variables sum to the same total 
propensity. 

In order for the results of propensity methods to be valid, it is important 
that the propensity scoring models include all relevant variables that could 
otherwise explain differences in both exposure and outcomes. Propensity 
score models can adjust only for confounding variables that are observed 
and measured. In other words, they are built upon the assumption that all 
relevant variables are measured and controlled for. If an important 
variable is omitted from the propensity model, the results of analyses may 
be affected. Westat made reasonable attempts to identify and control for a 
variety of confounding variables, include them in its models, and reduce 
bias. 
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EXECUTIVE OFFICE OF THE PRESIDENT 

OFFICE OF NATIONAL DRUG CONTROL POLICY 
Washington, D.C. 20503 

August 10, 2006 



Mr. David M. Walker 
Comptroller General 
Government Accountability Office 
441 G Street, NW 
Washington, DC 20548 

Dear Mr. Walker: 

I am writing in response to your request for comments on the Government Accountability 
Office (GAO) report (GAO-06-818) entitled, “Contractor’s National Evaluation Does Not Find 
That the Youth Anti-Drug Media Campaign Was Effective in Reducing Youth Drug Use.” I 
appreciate the opportunity to respond to the findings and recommendation made in the report and 
to provide you with additional information on how the Office of National Drug Control Policy is 
dedicated to the task of reducing youth drug use through media outreach. 

I have a number of concerns with the Westat findings and your report's assessment of 
them. In brief: 1) Westat’s evaluation is ill-suited to judge the impact of an ad campaign; 2) the 
findings are now more than 2 years old and have limited relevance; 3) conflicting evidence from 
other research is given minimal attention; 4) the Campaign has undergone major changes - with 
encouraging results to date; 5) our “due diligence” efforts to address the potential for hann are 
not well characterized; and 6) your recommendation to Congress offers insufficient detail to 
demonstrate satisfactory evidence of progress. Finally, I have identified probable consequences 
of further cuts to the Campaign budget that might be made pursuant to your recommendation to 
Congress. 

Westat Evaluation is Ill-Suited to Judging Impact of an Advertising Campaign 

Major advertisers, who spend billions of dollars annually on advertising in the U.S. and 
abroad, do not attempt to establish a causal relationship between advertising exposure and 
product sales, but evaluate the success of their advertising campaigns by rigorous testing of 
individual ads prior to air (which the Campaign has consistently done with increasing rigor), 
monitoring the performance of the ads once aired (which the Campaign has done) and by 
carefully developing correlations between various advertising messages, levels of media 
expenditure, and consumer attitudes and behavior. 

Establishing a causal relationship between exposure and outcomes is something major 
marketers rarely attempt because it is virtually impossible to do - particularly if both a pre- 
advertising baseline and / or unexposed control groups are lacking. As a senior market 
researcher at a major advertiser has recently said, “Even when campaign design includes a media 
blackout region as a control (inappropriate for a public-service campaign), non-measured factors 
make it impossible to isolate the effect of a single advertisement or advertising campaign.” 
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This is one reason why the “truth” anti-tobacco advertising campaign, acclaimed as a 
successful initiative in view of the significant declines we’ve seen in teen smoking, did not claim 
to prove a causal relationship between campaign exposure and smoking outcomes, reporting 
instead that the campaign was associated with substantial declines in youth smoking. They 
relied instead on the correlations between campaign weight levels in various markets (they were 
able to vary advertising weight by market) and teen behavior in those markets as tracked by the 
University of Michigan’s “Monitoring the Future” survey (see American Journal of Public 
Health, March 2005, “Evidence of a Dose-Response Relationship Between Antismoking Ads 
and Youth Smoking Prevalence”). 

Consequently, we take issue with the fundamental method pursued by Westat and GAO, 
and therefore, believe that the study’s findings are deeply flawed. 

Findings are “Old News” 

Your report’s findings do not come as a surprise to us, for each year as preliminary 
results have been made known to us, to Congress, and to the public through media coverage, we 
have studied the reports and have sought to improve the performance of the Campaign, using 
these and other data that we have available. We have responded when such findings, although 
not conclusive, were released to the public or to researchers, and have dealt with criticisms of the 
Campaign from adversaries, including those who advocate the legalization of drugs. And we 
have periodically needed to place these findings in context, especially because all major youth 
surveys report declining teen drug use, including Monitoring the Future, which documents a 19% 
decline in current illegal drug use among 8 th , 10 th and 12 th graders combined over the past four 
years, and a 21% decline in marijuana use from 1998 to 2005 among these youth. 

Conflicting Evidence from Other Research 

Although two recent studies that report findings contrary to Westat (Longshore and 
Palmgreen) are cited, no assessment of their importance is provided. GAO does not explain why 
such studies are not given more credence. For example, Longshore examined the possibility of 
synergistic effects between in-school drug education and the Campaign, and concludes that 
results “. . .showed that marijuana use in the past month was significantly less likely among 
adolescents who received both the ALERT Plus curriculum and weekly exposure to the 
Campaign’s anti-drug media messages.” This has significant implications nationally, and should 
not be dismissed. Palmgreen et al, whose paper has now been accepted for publication in the 
prestigious peer-reviewed American Journal of Public Health, concluded that the Campaign’s 
marijuana initiative “reversed upward developmental trends in high sensation seeking 30-day 
marijuana use and significantly reduced positive marijuana attitudes and beliefs in this at-risk 
population.” 

Another recent peer-reviewed journal article (Slater et al, “Combining in-school and 
community-based media efforts: reducing marijuana and alcohol uptake among younger 
adolescents,” in Health Education Research - Theory and Practice, September 2005) which 
evaluated the results of an in-school media campaign in 16 cities (8 campaign cities and 8 control 
communities) with a similar strategy and brand to our new Above the Influence campaign, has 
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shown very positive results. The authors conclude that “substance use uptake for youth in 
treatment communities was half or less than that of control communities” and that “effectiveness 
did not depend on the presence of an in-school prevention curriculum.” 

Your analysis thoughtfully acknowledges the serious challenges that Westat faced, due to 
a lack of a baseline and control group, the difficulties of retaining youth and parents in the study 
over the years, and the need to sort out other possible influences. These complexities add support 
to your expressed view that . .virtually all social science research is imperfect.” 

Westat, in its assessment of why the Campaign might potentially have contributed to 
youth experimentation with marijuana, seems to rely on the argument that frequent exposure to 
the Campaign’s anti-drug messages leads youth to conclude that most youth are, in fact, using 
marijuana, and in an effort to “fit in” with their peers, they decide to try marijuana. No 
theoretical basis for such an argument is provided in your report, nor is there any critique of the 
Campaign’s underlying theory, the well-tested Theory of Reasoned Action. Therefore, this 
reported effect is wholly counterintuitive because, by Westat data, it is clear that youth are seeing 
the Campaign's anti-drug ads, seeing them frequently, recalling and assessing them evermore 
favorably, holding ever-stronger anti-drug attitudes, receiving fewer offers of marijuana, and 
overall using drugs (including marijuana) less and less. No competing explanation for the 
substantial downturn in teen drug use or increase in anti-drug attitudes is offered. 

Further, key studies, including both Monitoring the Future (MTF) and the Partnership 
Attitude Tracking Study (PATS -- conducted by Roper for the Partnership for a Drug-Free 
America) report positive attitude changes. PATS data reveal that in the past few years more 
youth are likely to say most teens don’t smoke marijuana, and those who say they have friends 
who smoked marijuana declined substantially. Further, according to MTF, teens are increasingly 
likely to disapprove of trying marijuana, and these higher rates of disapproval are associated with 
lower rates of current use, especially among 10 th graders, the core target audience of the 
Campaign. And in an analysis conducted of National Survey of Drug Use and Health (NSDUH) 
data, youth who reported having seen or heard media prevention messages in the past year were 
significantly less likely to report illicit drug use. And some states and communities are 
observing that as teen drug use is declining, student surveys report an increase in the number of 
youth who say their peers disapprove of drug use and attribute the good news in part to anti-drug 
media messages (Coalition for a Drug Free Greater Cincinnati). Your report makes no claims 
that other media campaigns, or in-school drug education (which youth say they are getting less 
and less of) or other influences are responsible for these substantial shifts in beliefs, attitudes, 
and behaviors. 

“Due Diligence” Steps Taken to Remedy Potential Problems 

We have taken extensive “due diligence” steps which are briefly noted in your report. 

The report, however, fails to acknowledge the thoroughness of our actions to identify, assess, and 
attenuate any possible negative consequences of the campaign once Westat reported the 
possibility of such an effect. We have convened experts from mass communication, youth drug 
prevention, advertising, behavioral research, and related fields to explore what might be the 
theoretical basis for such potential effects. We have implemented recommendations to 
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strengthen qualitative and quantitative research methods to probe for such effects in subsequent 
advertising, and have made continual refinements in message strategy and have developed 
protocols for minimizing possible normative perceptions. We have increased the sensitivity to 
such a possibility among advertising planners of the Partnership for Drug-Free America - which 
under our direction provides most of the advertising for the campaign - as well as the pro bono 
advertising agencies who contribute their creative talents to the campaign. Further, we have 
strengthened our ability to detect any such effects through our monthly tracking studies which 
monitor youth responses to the media messages they are seeing and hearing. 

Major Changes in Campaign 

The Campaign is now substantially different from that which was measured when the last 
wave of data was collected by Westat more than two years ago. The Campaign has changed 
direction to be even more relevant to today’s youth, and after a full year of research, last 
November launched a new brand, “Above the Influence.” Designed to encourage youth to aspire 
to their full potential by avoiding the negative influences in their lives - specifically drugs and 
peer influences to use drugs - this new brand already is showing positive results in brand 
awareness (exceeding within six months the awareness of the previous brand, the Anti-Drug), 
perception of the risk in using drugs, and anti-drug attitudes, as revealed by monthly tracking 
surveys that are typical of advertising industry “best practices” for measuring an advertising 
campaign. This new Campaign has the breadth to address not only marijuana but other emerging 
drug threats, including methamphetamine and non-medical use of prescription drugs and over- 
the-counter medicines. We are pleased to report that “Above the Influence” already has won 
significant advertising and communications industry awards, including the prestigious Media 
Week “Media Plan of the Year,” for campaigns spending $25 million or more, the American 
Association of Advertising Agencies Jay Chiat Planning Award (the top honor for creative 
planning and strategy in its category), and two Webby Awards, including Best Youth Web Site 
of 2006, given by the International Academy of Digital Arts and Sciences. 

An Alternate Explanation for Counterintuitive Results 

A possible explanation for these counterintuitive findings could be a function of what is 
theorized by Westat to occur with heavy exposure to anti-drug messages. There is growing 
evidence from research in psychology and consumer behavior that asking people a question 
about their future behavior influences the subsequent performance of that very behavior, known 
as the “mere measurement effect.” The Westat evaluation, by conducting an extensive interview 
of youth (and their parents) in their homes, including showing them the ads being studied on as 
many as four separate occasions over the life of the evaluation, could stimulate interest in drugs 
where none previously existed, increase beliefs that drugs are important to youth and that more 
teens are using drags, thus stimulating interest in and causing intent to use drags and eventual 
drug experimentation. Impressionable youth, being inherently interested in pursuing behaviors 
that they believe would make them more grown-up, arguably would be more sensitive than 
adults to the effects of the evaluation interviews. One recent peer-reviewed journal article, for 
example, reported that “. . .when a question is asked about a socially non-normative health 
behavior (i.e. illegal drags), instead of decreases in the behavior we see increased rates of the 
non-normative behavior” (Williams, Fitzsimons and Block, “Simply asking questions about 
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health behavior increases both healthy and unhealthy behavior,” Social Influence, 2006). Such a 
problem would be less likely with a study design that used only cross-sectional surveys, but the 
frequency and intensity of youth interviews required by the Westat longitudinal design clearly 
would result in a greater impact of this type on survey respondents. 

Consequences of Delayed Report 

The long delay in receiving the GAO assessment of the Westat report has had serious 
operational consequences, such as severely compromising our ability to make progress on the 
next round of evaluation. More than two years ago, ONDCP announced an RFP for the new 
outcome evaluation, but in subsequent discussions GAO strongly discouraged us from pursuing 
that course because of the difficulty of evaluating this Campaign. GAO even posed a probable 
recommendation that we should first undertake an evaluability assessment. That 
recommendation is now not put forward by GAO. As a result, our plans for evaluation when 
resumed after more than two years will not provide meaningful outcome evaluation for several 
years to come. Meanwhile, we continue to use monthly tracking surveys to monitor Campaign 
progress, and although such studies cannot give us long term outcome data, they do permit real- 
time tracking of performance and allow for effective decision-making. 

Your Recommendation 

Finally, we are puzzled as to the lack of recommendations for improvement of the 
Campaign, given the GAO’s extensive review of the Westat results. For example, after the GAO 
staff briefing more than a year ago, we anticipated specific recommendations on future 
Campaign evaluations, or on Campaign theory, design, or implementation. While we appreciate 
your understanding that the Campaign has rigorously applied enonnous expertise to the 
Campaign and has made continual improvements, we are concerned with your conclusion that 
the result as judged by Westat “. . .raises questions concerning the understanding of the factors 
that are most salient to teens’ decision making about drugs and how they can be used to foster 
anti-drug decisions.” In addition, while your recommendation to Congress is clear, there are no 
criteria set forth for or even a cursory discussion of how one might demonstrate “. . .credible 
evidence of the effectiveness of the campaign on drug use outcomes” especially given the 
widely-accepted understanding that a media campaign, in and of itself, should not be held 
uniquely responsible for reductions in teen drug use. 

Consequences of Further Budget Cuts 

I want to make it clear that the consequences of further budget cuts to the campaign could 
have far-reaching and unfavorable consequences. Lessons learned from relevant youth health 
behaviors include recent evidence on tobacco use among teens which reveals that the downturn 
in media exposure to anti-tobacco messages already is resulting in decreased perception of risk, 
which more than 30 years of research shows is directly related to smoking rates (the same is true 
for illicit drugs, including marijuana). The 2005 MTF reported that all three grades surveyed 
showed a decline in weekly exposure to anti-tobacco messages, and the rate of decline of 
smoking is slowing and in fact has halted among 8 th graders, who have been the bellwethers of 
smoking trends among teens. Previous studies have established a relationship between exposure 
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to anti-tobacco messages and smoking rates among teens. CDC has observed that smoking 
prevention media campaigns are effective in reducing youth smoking initiation and has 
expressed concern that less exposure to such messages may translate to a reversal of the long 
downward trends in youth tobacco use. We should expect similar results for illicit drug use if 
anti-drug messages decline, and, in fact, already the MTF in 2005 has detected among 8 th graders 
a flattening out in the previously increasing perception of risk, due to somewhat lower media 
exposures. 

Further, because mass media and popular culture convey a pervasive and disheartening 
array of pro-drug messages that reach the nation’s youth, further cutting back the Campaign 
would have the unfortunate result of essentially abandoning our ability to counter those messages 
with clear, consistent, credible anti-drug messages, which only a national anti-drug media 
campaign can do. Teens today are exposed to media for well more than six hours per day. Half 
of teens live in households where there are no rules about TV exposure, and among the other half 
only one of five reports that those mles are actually enforced. Many of the most popular films 
among teens include scenes and references to drug use, seldom with any portrayal of social 
disapproval or negative consequences. Studies of websites accessible to any youth with 
computer access report ubiquitous — and highly detailed — pro-drug content, ranging from where 
to buy marijuana seeds, how to beat a drug test, and how to make methamphetamine. And 
anyone familiar with current trends in blogs and personal-space websites knows of the 
overwhelming pro-drug content, readily accessible to teens without their parents having any 
knowledge of such troubling exposures or risks to their children. 

Failure to continue the Campaign’s efforts to counter such messaging is the equivalent of 
raising a white flag to those who favor drug legalization, with the expectation that youth drug use 
soon would begin to rise, reversing years of hard-earned positive news. Compounding the 
problem is the fact that the news media seldom cover the risks of drug use to teens, but often 
report on the use of drugs by celebrities and others in the public eye, as well as efforts to legalize 
drugs and promote so-called medical marijuana. 

In addition, further cuts to the Campaign likely would create a chilling effect on those 
media companies who have donated well over a billion dollars worth of media time and space to 
ensure youth receive anti-drug messages, as well as those ad agencies - more than 80 to date - 
who have contributed their creative talents towards reducing teen drug use. The Partnership for a 
Drug-Free America, whose efforts helped persuade the Congress of the need for this Campaign, 
has been singularly effective at recruiting private sector largesse to this important goal. Such 
resources are essential to the success of our efforts. 

Conclusion 

As always, I remain interested in finding ways to improve the performance of the 
Campaign, as well as other efforts within the scope of the National Drug Control Strategy. I, 
too, once was skeptical of the ability of a media campaign to make a difference. Due to our 
considerable energies spent to re- focus and strengthen the Campaign, I have come to believe it is 
among the most important tools we have to reduce teen drug use. Much is at stake and we must 
work together to overcome the unwarranted cynicism over whether we can reduce drug use 
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